Attachment 1445827 Details for Bug 1581309 – /var/log/ovirt-engine/engine.log

/var/log/ovirt-engine/engine.log

engine.log (text/plain), 9.76 MB, created by Dave on 2018-05-30 12:28:04 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Dave

Created: 2018-05-30 12:28:04 UTC

Size: 9.76 MB

patch

obsolete

>2018-05-29 03:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1a82e3f5] Autorecovering 2 hosts
>2018-05-29 03:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1a82e3f5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:10:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [32ad692] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [32ad692] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [32ad692] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [32ad692] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:10:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [32ad692] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: fff6fe5
>2018-05-29 03:10:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [32ad692] FINISH, SetVdsStatusVDSCommand, log id: fff6fe5
>2018-05-29 03:10:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [32ad692] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:10:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [32ad692] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:10:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [32ad692] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:10:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [25ecb086] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:10:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [25ecb086] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:10:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [25ecb086] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:10:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [25ecb086] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:10:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [25ecb086] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1383380f
>2018-05-29 03:10:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [25ecb086] FINISH, SetVdsStatusVDSCommand, log id: 1383380f
>2018-05-29 03:10:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [25ecb086] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2b53e289
>2018-05-29 03:10:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [25ecb086] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2b53e289
>2018-05-29 03:10:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [25ecb086] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:10:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [25ecb086] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:10:02,116-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5cf7c216] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 592dc26d
>2018-05-29 03:10:02,118-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5cf7c216] FINISH, GetHardwareInfoVDSCommand, log id: 592dc26d
>2018-05-29 03:10:02,129-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3c7d7aee] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 34e61635
>2018-05-29 03:10:02,131-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3c7d7aee] FINISH, GetHardwareInfoVDSCommand, log id: 34e61635
>2018-05-29 03:10:02,132-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [12b03804] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:10:02,142-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [71cd273b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:10:02,146-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [39abab60] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:10:02,147-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1649b54b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:10:02,151-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [8780df1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:10:02,156-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [2a4ddb1d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:10:02,160-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1649b54b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6892272c
>2018-05-29 03:10:02,210-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [2a4ddb1d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 603ec9ee
>2018-05-29 03:10:02,584-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2a4ddb1d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:10:02,584-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [2a4ddb1d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 603ec9ee
>2018-05-29 03:10:02,588-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2a4ddb1d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:10:02,588-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [2a4ddb1d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:10:02,592-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2a4ddb1d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:10:02,593-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [2a4ddb1d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:10:02,597-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2a4ddb1d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:10:02,598-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [2a4ddb1d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:10:02,601-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [2a4ddb1d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:10:02,604-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2a4ddb1d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2a4ddb1d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:10:02,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 73e9bac0
>2018-05-29 03:10:02,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:10:02,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:10:02,708-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:10:02,708-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:10:02,708-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:10:02,708-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 73e9bac0
>2018-05-29 03:10:02,708-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:10:02,721-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [532215ab] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:10:02,722-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [532215ab] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1e9028af
>2018-05-29 03:10:02,724-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [532215ab] FINISH, SetVdsStatusVDSCommand, log id: 1e9028af
>2018-05-29 03:10:02,732-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [532215ab] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 532215ab, Job ID: 68b38be4-5559-46d4-a3ce-9f2a20a3f770, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:10:02,737-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [532215ab] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 8780df1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:10:02,739-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [532215ab] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 8780df1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:10:02,750-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [52244775] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:10:02,751-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [52244775] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:10:03,489-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1649b54b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:10:03,490-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1649b54b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6892272c
>2018-05-29 03:10:03,493-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1649b54b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:10:03,494-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1649b54b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:10:03,498-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1649b54b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:10:03,499-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1649b54b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:10:03,502-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1649b54b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:10:03,503-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1649b54b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:10:03,506-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1649b54b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:10:03,509-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1649b54b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1649b54b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:10:03,511-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2063e584
>2018-05-29 03:10:03,511-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:10:03,610-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:10:03,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:10:03,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:10:03,610-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:10:03,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2063e584
>2018-05-29 03:10:03,610-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:10:03,623-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [52c2d1fa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:10:03,624-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [52c2d1fa] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: eedb648
>2018-05-29 03:10:03,626-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [52c2d1fa] FINISH, SetVdsStatusVDSCommand, log id: eedb648
>2018-05-29 03:10:03,631-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [52c2d1fa] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 52c2d1fa, Job ID: 598771d5-a239-409e-a12f-169ce7e9a37e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:10:03,636-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [52c2d1fa] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 71cd273b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:10:03,638-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [52c2d1fa] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 71cd273b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:10:03,649-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [3d9612e1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:10:03,650-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [3d9612e1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [19c413c] Autorecovering 2 hosts
>2018-05-29 03:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [19c413c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:15:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7c2fb44f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:15:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7c2fb44f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7c2fb44f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7c2fb44f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:15:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7c2fb44f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 745b3af9
>2018-05-29 03:15:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7c2fb44f] FINISH, SetVdsStatusVDSCommand, log id: 745b3af9
>2018-05-29 03:15:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7c2fb44f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:15:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7c2fb44f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:15:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [7c2fb44f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:15:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [35706c0e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:15:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [35706c0e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:15:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [35706c0e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:15:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [35706c0e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:15:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [35706c0e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f50fc51
>2018-05-29 03:15:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [35706c0e] FINISH, SetVdsStatusVDSCommand, log id: f50fc51
>2018-05-29 03:15:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [35706c0e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4820c29
>2018-05-29 03:15:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [35706c0e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4820c29
>2018-05-29 03:15:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [35706c0e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:15:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [35706c0e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:15:00,399-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3280d2e4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4d1700f8
>2018-05-29 03:15:00,401-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3280d2e4] FINISH, GetHardwareInfoVDSCommand, log id: 4d1700f8
>2018-05-29 03:15:00,413-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6afb5d72] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:15:00,422-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3a9a5cd0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:15:00,426-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [68dd6d79] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:15:00,438-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [68dd6d79] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3e15e54d
>2018-05-29 03:15:00,815-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68dd6d79] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:15:00,815-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [68dd6d79] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3e15e54d
>2018-05-29 03:15:00,819-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68dd6d79] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:15:00,820-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [68dd6d79] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:15:00,824-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68dd6d79] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:15:00,824-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [68dd6d79] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:15:00,828-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68dd6d79] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:15:00,828-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [68dd6d79] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:15:00,832-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [68dd6d79] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:15:00,834-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68dd6d79] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 68dd6d79, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:15:00,841-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 61bd51c9
>2018-05-29 03:15:00,841-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:15:00,937-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:15:00,937-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:15:00,937-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:15:00,937-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:15:00,937-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 61bd51c9
>2018-05-29 03:15:00,937-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:15:00,949-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [59b863b6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:15:00,950-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [59b863b6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 16dcbd21
>2018-05-29 03:15:00,953-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [59b863b6] FINISH, SetVdsStatusVDSCommand, log id: 16dcbd21
>2018-05-29 03:15:00,985-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [59b863b6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 59b863b6, Job ID: acecd04a-e3ba-423f-88ee-3ffbe5dde0bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:15:00,990-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [59b863b6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3a9a5cd0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:15:00,992-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [59b863b6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3a9a5cd0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:15:01,003-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [6e914885] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:15:01,004-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [6e914885] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:15:01,334-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [293b14b1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4be83a8a
>2018-05-29 03:15:01,336-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [293b14b1] FINISH, GetHardwareInfoVDSCommand, log id: 4be83a8a
>2018-05-29 03:15:01,348-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [107779bd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:15:01,357-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1fe383a1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:15:01,363-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [3c3402da] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:15:01,375-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [3c3402da] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7435d719
>2018-05-29 03:15:01,754-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c3402da] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:15:01,754-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [3c3402da] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7435d719
>2018-05-29 03:15:01,758-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c3402da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:15:01,759-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3c3402da] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:15:01,763-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c3402da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:15:01,763-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3c3402da] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:15:01,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c3402da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:15:01,768-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3c3402da] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:15:01,771-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [3c3402da] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:15:01,774-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c3402da] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3c3402da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:15:01,777-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 29396ef1
>2018-05-29 03:15:01,777-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:15:01,871-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:15:01,871-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:15:01,871-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:15:01,871-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:15:01,871-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 29396ef1
>2018-05-29 03:15:01,871-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:15:01,884-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [3c098215] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:15:01,885-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3c098215] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4e564a15
>2018-05-29 03:15:01,887-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3c098215] FINISH, SetVdsStatusVDSCommand, log id: 4e564a15
>2018-05-29 03:15:01,893-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c098215] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c098215, Job ID: 87b1c92e-6918-4378-bc25-dfdf4747365c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:15:01,897-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c098215] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1fe383a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:15:01,899-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c098215] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1fe383a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:15:01,909-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2e7367c3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:15:01,911-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2e7367c3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1430a203] Autorecovering 2 hosts
>2018-05-29 03:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1430a203] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:20:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [734f5e84] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:20:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [734f5e84] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:20:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [734f5e84] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:20:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [734f5e84] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:20:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [734f5e84] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25699886
>2018-05-29 03:20:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [734f5e84] FINISH, SetVdsStatusVDSCommand, log id: 25699886
>2018-05-29 03:20:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [734f5e84] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:20:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [734f5e84] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:20:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [734f5e84] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:20:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61fae362] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:20:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61fae362] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:20:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61fae362] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:20:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61fae362] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:20:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [61fae362] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4a5c53c1
>2018-05-29 03:20:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [61fae362] FINISH, SetVdsStatusVDSCommand, log id: 4a5c53c1
>2018-05-29 03:20:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [61fae362] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1293a913
>2018-05-29 03:20:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [61fae362] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1293a913
>2018-05-29 03:20:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61fae362] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:20:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61fae362] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:20:01,645-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [17bfd319] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 44aeb9e0
>2018-05-29 03:20:01,647-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [17bfd319] FINISH, GetHardwareInfoVDSCommand, log id: 44aeb9e0
>2018-05-29 03:20:01,661-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [44e66103] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:20:01,666-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [420ed963] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:20:01,671-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [505fba9a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:20:01,678-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [505fba9a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1a09a879
>2018-05-29 03:20:02,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [505fba9a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:20:02,052-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [505fba9a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1a09a879
>2018-05-29 03:20:02,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [505fba9a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:20:02,057-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [505fba9a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:20:02,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [505fba9a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:20:02,061-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [505fba9a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:20:02,065-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [505fba9a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:20:02,066-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [505fba9a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:20:02,069-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [505fba9a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:20:02,072-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [505fba9a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 505fba9a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:20:02,078-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 515ac58c
>2018-05-29 03:20:02,078-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:20:02,176-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:20:02,176-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:20:02,176-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:20:02,177-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:20:02,177-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 515ac58c
>2018-05-29 03:20:02,177-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:20:02,190-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [2be48ce] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:20:02,191-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2be48ce] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f56856c
>2018-05-29 03:20:02,194-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2be48ce] FINISH, SetVdsStatusVDSCommand, log id: 5f56856c
>2018-05-29 03:20:02,225-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2be48ce] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2be48ce, Job ID: e4993c48-fbb5-41d4-bdd3-31140e9f51bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:20:02,230-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2be48ce] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 420ed963, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:20:02,233-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2be48ce] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 420ed963, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:20:02,244-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [5e54b9fe] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:20:02,245-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [5e54b9fe] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:20:02,577-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [68c413ab] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3545f278
>2018-05-29 03:20:02,579-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [68c413ab] FINISH, GetHardwareInfoVDSCommand, log id: 3545f278
>2018-05-29 03:20:02,591-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [707aea05] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:20:02,596-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [63d97c29] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:20:02,601-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [76dd858f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:20:02,607-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [76dd858f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: fca1ebf
>2018-05-29 03:20:02,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [76dd858f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:20:02,987-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [76dd858f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: fca1ebf
>2018-05-29 03:20:02,991-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [76dd858f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:20:02,991-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [76dd858f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:20:02,995-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [76dd858f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:20:02,996-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [76dd858f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:20:03,000-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [76dd858f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:20:03,000-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [76dd858f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:20:03,004-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [76dd858f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:20:03,006-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [76dd858f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 76dd858f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:20:03,009-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 89f9187
>2018-05-29 03:20:03,009-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:20:03,105-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:20:03,105-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:20:03,105-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:20:03,105-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:20:03,105-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 89f9187
>2018-05-29 03:20:03,105-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:20:03,118-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [15544d6f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:20:03,119-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [15544d6f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2cf0ab3b
>2018-05-29 03:20:03,122-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [15544d6f] FINISH, SetVdsStatusVDSCommand, log id: 2cf0ab3b
>2018-05-29 03:20:03,127-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [15544d6f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 15544d6f, Job ID: 62c86305-040e-4983-a0e7-ac400a2cdfe5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:20:03,132-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [15544d6f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 63d97c29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:20:03,134-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [15544d6f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 63d97c29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:20:03,144-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [7792a274] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:20:03,145-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7792a274] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5d7c9b91] Autorecovering 2 hosts
>2018-05-29 03:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5d7c9b91] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:25:00,036-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [113e6344] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:25:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [113e6344] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:25:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [113e6344] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:25:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [113e6344] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:25:00,045-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [113e6344] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32580ed5
>2018-05-29 03:25:00,047-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [113e6344] FINISH, SetVdsStatusVDSCommand, log id: 32580ed5
>2018-05-29 03:25:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [113e6344] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:25:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [113e6344] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:25:00,051-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [113e6344] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:25:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4d425e68] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:25:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4d425e68] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:25:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4d425e68] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:25:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4d425e68] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:25:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4d425e68] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4d4549f
>2018-05-29 03:25:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4d425e68] FINISH, SetVdsStatusVDSCommand, log id: 4d4549f
>2018-05-29 03:25:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [4d425e68] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6a0257e6
>2018-05-29 03:25:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [4d425e68] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6a0257e6
>2018-05-29 03:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4d425e68] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:25:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4d425e68] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:25:00,713-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [2ac08dd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 595695bf
>2018-05-29 03:25:00,715-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [2ac08dd] FINISH, GetHardwareInfoVDSCommand, log id: 595695bf
>2018-05-29 03:25:00,727-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [90daa54] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:25:00,735-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6a58e8d1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:25:00,741-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1ce8f94c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:25:00,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1ce8f94c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 79c64c9
>2018-05-29 03:25:01,133-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1ce8f94c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:25:01,133-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1ce8f94c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 79c64c9
>2018-05-29 03:25:01,137-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1ce8f94c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:25:01,138-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1ce8f94c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:25:01,142-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1ce8f94c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:25:01,142-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1ce8f94c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:25:01,146-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1ce8f94c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:25:01,147-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1ce8f94c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:25:01,150-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1ce8f94c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:25:01,153-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1ce8f94c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1ce8f94c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:25:01,159-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 43b6df3
>2018-05-29 03:25:01,159-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:25:01,257-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:25:01,257-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:25:01,257-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:25:01,257-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:25:01,257-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 43b6df3
>2018-05-29 03:25:01,257-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:25:01,271-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [5ed20c64] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:25:01,272-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5ed20c64] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a8b7783
>2018-05-29 03:25:01,274-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5ed20c64] FINISH, SetVdsStatusVDSCommand, log id: 5a8b7783
>2018-05-29 03:25:01,306-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5ed20c64] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5ed20c64, Job ID: 61352822-d8a9-4823-9351-ee7798c4c6f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:25:01,311-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5ed20c64] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6a58e8d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:25:01,313-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5ed20c64] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6a58e8d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:25:01,323-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [48c2aa82] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:25:01,324-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [48c2aa82] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:25:02,954-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2b09c245] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 42b10bda
>2018-05-29 03:25:02,956-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2b09c245] FINISH, GetHardwareInfoVDSCommand, log id: 42b10bda
>2018-05-29 03:25:02,969-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [4625a336] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:25:02,973-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [282a4647] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:25:02,978-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6463d999] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:25:02,985-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6463d999] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 357b462a
>2018-05-29 03:25:03,359-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6463d999] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:25:03,360-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6463d999] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 357b462a
>2018-05-29 03:25:03,363-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6463d999] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:25:03,364-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6463d999] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:25:03,369-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6463d999] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:25:03,369-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6463d999] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:25:03,373-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6463d999] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:25:03,374-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6463d999] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:25:03,377-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6463d999] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:25:03,379-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6463d999] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6463d999, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:25:03,382-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: cb8f171
>2018-05-29 03:25:03,382-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:25:03,479-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:25:03,479-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:25:03,479-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:25:03,479-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:25:03,479-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: cb8f171
>2018-05-29 03:25:03,479-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:25:03,492-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [906012a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:25:03,493-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [906012a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41c4a241
>2018-05-29 03:25:03,496-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [906012a] FINISH, SetVdsStatusVDSCommand, log id: 41c4a241
>2018-05-29 03:25:03,501-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [906012a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 906012a, Job ID: 99b294be-481c-409b-844a-d3f607302c87, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:25:03,505-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [906012a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 282a4647, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:25:03,507-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [906012a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 282a4647, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:25:03,518-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [7bc980c3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:25:03,519-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7bc980c3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6a5a91] Autorecovering 2 hosts
>2018-05-29 03:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6a5a91] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:30:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6bf710b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:30:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6bf710b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:30:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6bf710b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:30:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6bf710b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:30:00,047-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6bf710b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5ead38e9
>2018-05-29 03:30:00,049-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6bf710b] FINISH, SetVdsStatusVDSCommand, log id: 5ead38e9
>2018-05-29 03:30:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6bf710b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:30:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6bf710b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:30:00,051-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6bf710b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:30:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6c52b27a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:30:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6c52b27a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:30:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6c52b27a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:30:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6c52b27a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:30:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6c52b27a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 347fc78
>2018-05-29 03:30:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6c52b27a] FINISH, SetVdsStatusVDSCommand, log id: 347fc78
>2018-05-29 03:30:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [6c52b27a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 102bb8d8
>2018-05-29 03:30:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [6c52b27a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 102bb8d8
>2018-05-29 03:30:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6c52b27a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6c52b27a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:30:01,262-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [645a275e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1448299a
>2018-05-29 03:30:01,263-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [645a275e] FINISH, GetHardwareInfoVDSCommand, log id: 1448299a
>2018-05-29 03:30:01,276-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [60b9924d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:30:01,285-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [28beddc0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:30:01,290-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6f1b0859] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:30:01,302-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6f1b0859] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7b718665
>2018-05-29 03:30:01,676-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6f1b0859] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:30:01,676-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6f1b0859] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7b718665
>2018-05-29 03:30:01,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6f1b0859] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:30:01,680-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6f1b0859] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:30:01,684-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6f1b0859] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:30:01,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6f1b0859] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:30:01,688-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6f1b0859] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:30:01,689-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6f1b0859] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:30:01,692-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6f1b0859] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:30:01,694-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6f1b0859] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6f1b0859, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:30:01,700-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7682432e
>2018-05-29 03:30:01,700-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:30:01,799-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:30:01,799-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:30:01,799-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:30:01,799-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:30:01,799-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7682432e
>2018-05-29 03:30:01,799-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:30:01,812-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [1d4fd5f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:30:01,813-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1d4fd5f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2fa9a319
>2018-05-29 03:30:01,815-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1d4fd5f] FINISH, SetVdsStatusVDSCommand, log id: 2fa9a319
>2018-05-29 03:30:01,845-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d4fd5f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1d4fd5f, Job ID: c19db18f-86a7-4866-9784-4f0105b280d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:30:01,850-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d4fd5f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 28beddc0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:30:01,852-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d4fd5f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 28beddc0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:30:01,862-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [3a357293] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:30:01,863-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [3a357293] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:30:01,941-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [19bf3608] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1b1e6ccc
>2018-05-29 03:30:01,942-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [19bf3608] FINISH, GetHardwareInfoVDSCommand, log id: 1b1e6ccc
>2018-05-29 03:30:01,955-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [f8ed03b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:30:01,960-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [7d6558ff] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:30:01,965-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7caf5f5f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:30:01,971-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7caf5f5f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 53694b83
>2018-05-29 03:30:02,348-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7caf5f5f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:30:02,348-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7caf5f5f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 53694b83
>2018-05-29 03:30:02,352-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7caf5f5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:30:02,353-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7caf5f5f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:30:02,357-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7caf5f5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:30:02,358-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7caf5f5f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:30:02,361-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7caf5f5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:30:02,362-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7caf5f5f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:30:02,365-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7caf5f5f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:30:02,367-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7caf5f5f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7caf5f5f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:30:02,370-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4c10fc01
>2018-05-29 03:30:02,370-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:30:02,466-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:30:02,467-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:30:02,467-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:30:02,467-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:30:02,467-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4c10fc01
>2018-05-29 03:30:02,467-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:30:02,479-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [23264963] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:30:02,481-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [23264963] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1650244a
>2018-05-29 03:30:02,483-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [23264963] FINISH, SetVdsStatusVDSCommand, log id: 1650244a
>2018-05-29 03:30:02,488-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [23264963] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 23264963, Job ID: 29f42b5e-03b5-41f8-8a7a-d8fa131931cd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:30:02,493-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [23264963] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7d6558ff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:30:02,495-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [23264963] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7d6558ff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:30:02,505-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [76d2c60e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:30:02,506-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [76d2c60e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [2cfea204] Autorecovering 2 hosts
>2018-05-29 03:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [2cfea204] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:35:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3b655e80] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3b655e80] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3b655e80] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3b655e80] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:35:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3b655e80] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42c7815d
>2018-05-29 03:35:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3b655e80] FINISH, SetVdsStatusVDSCommand, log id: 42c7815d
>2018-05-29 03:35:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3b655e80] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:35:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3b655e80] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:35:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [3b655e80] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:35:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6d2e49bf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:35:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6d2e49bf] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:35:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6d2e49bf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:35:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6d2e49bf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:35:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6d2e49bf] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f7c23b7
>2018-05-29 03:35:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6d2e49bf] FINISH, SetVdsStatusVDSCommand, log id: f7c23b7
>2018-05-29 03:35:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [6d2e49bf] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 18b9c11c
>2018-05-29 03:35:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [6d2e49bf] FINISH, SetHaMaintenanceModeVDSCommand, log id: 18b9c11c
>2018-05-29 03:35:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6d2e49bf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:35:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6d2e49bf] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:35:02,478-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3d111d2b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3cb09756
>2018-05-29 03:35:02,480-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3d111d2b] FINISH, GetHardwareInfoVDSCommand, log id: 3cb09756
>2018-05-29 03:35:02,494-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [ec92e42] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:35:02,498-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [5fba9753] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:35:02,504-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [65cdcb27] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:35:02,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [65cdcb27] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 35d561e8
>2018-05-29 03:35:02,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65cdcb27] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:35:02,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [65cdcb27] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 35d561e8
>2018-05-29 03:35:02,890-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65cdcb27] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:35:02,891-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [65cdcb27] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:35:02,894-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65cdcb27] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:35:02,895-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [65cdcb27] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:35:02,899-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65cdcb27] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:35:02,899-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [65cdcb27] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:35:02,902-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [65cdcb27] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:35:02,905-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65cdcb27] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 65cdcb27, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:35:02,911-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1849615b
>2018-05-29 03:35:02,911-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:35:03,011-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:35:03,011-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:35:03,011-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:35:03,011-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:35:03,011-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1849615b
>2018-05-29 03:35:03,012-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:35:03,024-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [18dd62db] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:35:03,025-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [18dd62db] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 57ad74c5
>2018-05-29 03:35:03,027-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [18dd62db] FINISH, SetVdsStatusVDSCommand, log id: 57ad74c5
>2018-05-29 03:35:03,035-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18dd62db] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 18dd62db, Job ID: 4e3c72db-5531-42e9-9430-952f8ea3fc27, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:35:03,039-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18dd62db] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5fba9753, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:35:03,041-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18dd62db] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5fba9753, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:35:03,051-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [b4f5172] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:35:03,052-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [b4f5172] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:35:03,071-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [211fc20c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 515314e0
>2018-05-29 03:35:03,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [211fc20c] FINISH, GetHardwareInfoVDSCommand, log id: 515314e0
>2018-05-29 03:35:03,085-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3b6707ac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:35:03,090-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [228ed09b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:35:03,095-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3f1110e0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:35:03,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3f1110e0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5ade2201
>2018-05-29 03:35:03,476-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f1110e0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:35:03,476-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3f1110e0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5ade2201
>2018-05-29 03:35:03,479-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f1110e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:35:03,480-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3f1110e0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:35:03,484-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f1110e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:35:03,485-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3f1110e0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:35:03,489-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f1110e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:35:03,490-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3f1110e0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:35:03,493-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3f1110e0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:35:03,496-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f1110e0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3f1110e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:35:03,499-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5ad5b6f9
>2018-05-29 03:35:03,499-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:35:03,594-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:35:03,594-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:35:03,594-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:35:03,594-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:35:03,594-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5ad5b6f9
>2018-05-29 03:35:03,594-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:35:03,606-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [4ef54c2c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:35:03,608-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4ef54c2c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ac34a82
>2018-05-29 03:35:03,610-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4ef54c2c] FINISH, SetVdsStatusVDSCommand, log id: 3ac34a82
>2018-05-29 03:35:03,615-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4ef54c2c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4ef54c2c, Job ID: d90d2aff-d7b2-494c-9d6c-c560f0a7d9bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:35:03,620-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4ef54c2c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 228ed09b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:35:03,622-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4ef54c2c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 228ed09b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:35:03,632-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [3c0f857b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:35:03,634-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [3c0f857b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:35:35,001-04 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler6) [1ae7fcb9] Start deleteAgedOutAuditLogs
>2018-05-29 03:35:35,001-04 INFO  [org.ovirt.engine.core.bll.CommandEntityCleanupManager] (DefaultQuartzScheduler7) [2c942208] Start deleteAgedOutCommandEntities
>2018-05-29 03:35:35,002-04 INFO  [org.ovirt.engine.core.bll.CommandEntityCleanupManager] (DefaultQuartzScheduler7) [2c942208] Finished deleteAgedOutCommandEntities
>2018-05-29 03:35:35,002-04 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler6) [1ae7fcb9] Finished deleteAgedOutAuditLogs
>2018-05-29 03:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [61f1e891] Autorecovering 2 hosts
>2018-05-29 03:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [61f1e891] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:40:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1affe50d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:40:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1affe50d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:40:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1affe50d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:40:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1affe50d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:40:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1affe50d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b7f99b7
>2018-05-29 03:40:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1affe50d] FINISH, SetVdsStatusVDSCommand, log id: 3b7f99b7
>2018-05-29 03:40:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1affe50d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:40:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1affe50d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:40:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1affe50d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:40:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c2ff5c4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:40:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c2ff5c4] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:40:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c2ff5c4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:40:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c2ff5c4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:40:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5c2ff5c4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f586250
>2018-05-29 03:40:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5c2ff5c4] FINISH, SetVdsStatusVDSCommand, log id: 3f586250
>2018-05-29 03:40:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5c2ff5c4] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5f1dc209
>2018-05-29 03:40:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5c2ff5c4] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5f1dc209
>2018-05-29 03:40:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c2ff5c4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:40:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c2ff5c4] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:40:00,715-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [484c219c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 555a3ad6
>2018-05-29 03:40:00,717-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [484c219c] FINISH, GetHardwareInfoVDSCommand, log id: 555a3ad6
>2018-05-29 03:40:00,730-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [2bcf3fd3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:40:00,739-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [b882ec7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:40:00,744-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [e797d73] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:40:00,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [e797d73] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7ce1a2d0
>2018-05-29 03:40:01,171-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e797d73] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:40:01,171-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [e797d73] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7ce1a2d0
>2018-05-29 03:40:01,175-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e797d73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:40:01,175-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [e797d73] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:40:01,179-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e797d73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:40:01,180-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [e797d73] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:40:01,183-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e797d73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:40:01,184-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [e797d73] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:40:01,187-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [e797d73] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:40:01,190-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e797d73] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: e797d73, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:40:01,196-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: ac0d962
>2018-05-29 03:40:01,196-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:40:01,251-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [414cd06] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5cfa2b70
>2018-05-29 03:40:01,253-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [414cd06] FINISH, GetHardwareInfoVDSCommand, log id: 5cfa2b70
>2018-05-29 03:40:01,265-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3e1dcf3a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:40:01,270-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [51eebd9e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:40:01,274-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6af876a3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:40:01,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6af876a3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 54ac8e4d
>2018-05-29 03:40:01,295-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:40:01,295-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:40:01,295-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:40:01,295-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:40:01,295-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: ac0d962
>2018-05-29 03:40:01,295-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:40:01,308-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [1e16965c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:40:01,309-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1e16965c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 23587e3f
>2018-05-29 03:40:01,311-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1e16965c] FINISH, SetVdsStatusVDSCommand, log id: 23587e3f
>2018-05-29 03:40:01,344-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1e16965c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1e16965c, Job ID: 5fe4c767-9d75-4076-ad98-4b221fb730e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:40:01,349-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1e16965c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: b882ec7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:40:01,351-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1e16965c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: b882ec7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:40:01,361-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [142d51f6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:40:01,362-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [142d51f6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:40:01,657-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6af876a3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:40:01,657-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6af876a3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 54ac8e4d
>2018-05-29 03:40:01,661-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6af876a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:40:01,661-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6af876a3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:40:01,665-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6af876a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:40:01,666-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6af876a3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:40:01,669-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6af876a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:40:01,670-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6af876a3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:40:01,673-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6af876a3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:40:01,676-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6af876a3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6af876a3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:40:01,679-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4e2eb181
>2018-05-29 03:40:01,679-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:40:01,778-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:40:01,778-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:40:01,778-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:40:01,778-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:40:01,778-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4e2eb181
>2018-05-29 03:40:01,778-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:40:01,789-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [7fdf06c0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:40:01,790-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7fdf06c0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45098578
>2018-05-29 03:40:01,793-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7fdf06c0] FINISH, SetVdsStatusVDSCommand, log id: 45098578
>2018-05-29 03:40:01,798-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fdf06c0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7fdf06c0, Job ID: 793e88fe-41cf-4269-8dd8-4d27741359d7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:40:01,802-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fdf06c0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 51eebd9e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:40:01,804-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fdf06c0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 51eebd9e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:40:01,814-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [5ac0cec2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:40:01,815-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [5ac0cec2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [415bbb35] Autorecovering 2 hosts
>2018-05-29 03:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [415bbb35] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:45:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5848a29f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5848a29f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5848a29f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5848a29f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:45:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5848a29f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3412b4db
>2018-05-29 03:45:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5848a29f] FINISH, SetVdsStatusVDSCommand, log id: 3412b4db
>2018-05-29 03:45:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5848a29f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:45:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5848a29f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:45:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [5848a29f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:45:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d1b9a2e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d1b9a2e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:45:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d1b9a2e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:45:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d1b9a2e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:45:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1d1b9a2e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bfeea94
>2018-05-29 03:45:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1d1b9a2e] FINISH, SetVdsStatusVDSCommand, log id: 3bfeea94
>2018-05-29 03:45:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [1d1b9a2e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 726a2f62
>2018-05-29 03:45:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [1d1b9a2e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 726a2f62
>2018-05-29 03:45:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d1b9a2e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:45:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d1b9a2e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:45:02,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [482150c6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1c0391c3
>2018-05-29 03:45:02,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [482150c6] FINISH, GetHardwareInfoVDSCommand, log id: 1c0391c3
>2018-05-29 03:45:02,108-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [1791cefb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:45:02,117-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [743cb761] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:45:02,122-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [41ecf5e0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:45:02,134-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [41ecf5e0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 10310eb5
>2018-05-29 03:45:02,386-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [77e41bd6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5488bd3e
>2018-05-29 03:45:02,387-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [77e41bd6] FINISH, GetHardwareInfoVDSCommand, log id: 5488bd3e
>2018-05-29 03:45:02,401-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [1819ae3e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:45:02,405-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [74a2918] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:45:02,410-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [29bfd24f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:45:02,418-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [29bfd24f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 210160cc
>2018-05-29 03:45:02,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41ecf5e0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:45:02,508-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [41ecf5e0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 10310eb5
>2018-05-29 03:45:02,512-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41ecf5e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:45:02,513-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41ecf5e0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:45:02,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41ecf5e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:45:02,517-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41ecf5e0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:45:02,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41ecf5e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:45:02,522-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41ecf5e0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:45:02,524-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [41ecf5e0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:45:02,527-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41ecf5e0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 41ecf5e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:45:02,534-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 39417a0b
>2018-05-29 03:45:02,534-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:45:02,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:45:02,631-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:45:02,631-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:45:02,631-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:45:02,631-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: 39417a0b
>2018-05-29 03:45:02,631-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:45:02,646-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5e00da6c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:45:02,647-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5e00da6c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 278b8e29
>2018-05-29 03:45:02,649-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5e00da6c] FINISH, SetVdsStatusVDSCommand, log id: 278b8e29
>2018-05-29 03:45:02,684-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5e00da6c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5e00da6c, Job ID: d3243ab9-c89e-416c-a0c9-e879e2f70453, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:45:02,688-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5e00da6c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 743cb761, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:45:02,691-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5e00da6c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 743cb761, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:45:02,701-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [23c9c4b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:45:02,703-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [23c9c4b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:45:02,797-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29bfd24f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:45:02,797-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [29bfd24f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 210160cc
>2018-05-29 03:45:02,801-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29bfd24f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:45:02,802-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [29bfd24f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:45:02,806-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29bfd24f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:45:02,806-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [29bfd24f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:45:02,810-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29bfd24f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:45:02,811-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [29bfd24f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:45:02,814-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [29bfd24f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:45:02,817-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29bfd24f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 29bfd24f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:45:02,820-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5a219251
>2018-05-29 03:45:02,820-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:45:02,919-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:45:02,919-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:45:02,919-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:45:02,919-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:45:02,919-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5a219251
>2018-05-29 03:45:02,919-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:45:02,932-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [56550ca2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:45:02,933-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [56550ca2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5202e5ab
>2018-05-29 03:45:02,937-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [56550ca2] FINISH, SetVdsStatusVDSCommand, log id: 5202e5ab
>2018-05-29 03:45:02,942-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [56550ca2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 56550ca2, Job ID: 411bbbfd-bfbd-4732-a6c4-79a40ff85dd2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:45:02,947-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [56550ca2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 74a2918, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:45:02,949-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [56550ca2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 74a2918, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:45:02,960-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [67b0ab05] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:45:02,961-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [67b0ab05] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:50:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [121e3f7] Autorecovering 2 hosts
>2018-05-29 03:50:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [121e3f7] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:50:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [653f701c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:50:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [653f701c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:50:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [653f701c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:50:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [653f701c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:50:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [653f701c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76c7b545
>2018-05-29 03:50:00,046-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [653f701c] FINISH, SetVdsStatusVDSCommand, log id: 76c7b545
>2018-05-29 03:50:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [653f701c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:50:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [653f701c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:50:00,048-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [653f701c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:50:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [46ebd0f8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:50:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [46ebd0f8] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:50:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [46ebd0f8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:50:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [46ebd0f8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:50:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [46ebd0f8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7cb54f42
>2018-05-29 03:50:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [46ebd0f8] FINISH, SetVdsStatusVDSCommand, log id: 7cb54f42
>2018-05-29 03:50:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [46ebd0f8] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6dd069fa
>2018-05-29 03:50:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [46ebd0f8] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6dd069fa
>2018-05-29 03:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [46ebd0f8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:50:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [46ebd0f8] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:50:00,364-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5b8f5905] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5a2978b2
>2018-05-29 03:50:00,366-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5b8f5905] FINISH, GetHardwareInfoVDSCommand, log id: 5a2978b2
>2018-05-29 03:50:00,378-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [12408f96] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:50:00,387-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [5355cc17] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:50:00,392-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1c705773] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:50:00,450-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1c705773] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3e8cc25c
>2018-05-29 03:50:00,519-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [489399b2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 40e58b76
>2018-05-29 03:50:00,520-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [489399b2] FINISH, GetHardwareInfoVDSCommand, log id: 40e58b76
>2018-05-29 03:50:00,533-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [122e46a5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:50:00,537-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [137bb5c5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:50:00,543-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4675a3e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:50:00,549-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4675a3e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 435510d4
>2018-05-29 03:50:00,929-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4675a3e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:50:00,929-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4675a3e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 435510d4
>2018-05-29 03:50:00,933-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4675a3e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:50:00,934-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4675a3e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:50:00,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4675a3e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:50:00,938-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4675a3e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:50:00,942-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4675a3e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:50:00,943-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4675a3e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:50:00,946-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4675a3e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:50:00,949-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4675a3e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4675a3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:50:00,956-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2d3056af
>2018-05-29 03:50:00,956-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:50:01,051-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:50:01,051-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:50:01,051-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:50:01,051-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:50:01,051-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2d3056af
>2018-05-29 03:50:01,051-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:50:01,065-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [a3d86df] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:50:01,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [a3d86df] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 679ae284
>2018-05-29 03:50:01,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [a3d86df] FINISH, SetVdsStatusVDSCommand, log id: 679ae284
>2018-05-29 03:50:01,099-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a3d86df] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: a3d86df, Job ID: 34e9620c-584c-4fd6-81b8-8d01d5a896a3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:50:01,103-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a3d86df] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 137bb5c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:50:01,105-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a3d86df] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 137bb5c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:50:01,116-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [6430f565] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:50:01,117-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [6430f565] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:50:02,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c705773] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:50:02,767-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1c705773] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3e8cc25c
>2018-05-29 03:50:02,771-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c705773] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:50:02,772-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1c705773] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:50:02,776-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c705773] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:50:02,777-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1c705773] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:50:02,780-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c705773] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:50:02,781-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1c705773] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:50:02,785-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1c705773] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:50:02,787-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c705773] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1c705773, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:50:02,790-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: ce8ece
>2018-05-29 03:50:02,790-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:50:02,890-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:50:02,890-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:50:02,890-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:50:02,890-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:50:02,890-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: ce8ece
>2018-05-29 03:50:02,890-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:50:02,903-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1d5d8dca] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:50:02,904-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1d5d8dca] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 34010f76
>2018-05-29 03:50:02,907-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1d5d8dca] FINISH, SetVdsStatusVDSCommand, log id: 34010f76
>2018-05-29 03:50:02,912-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1d5d8dca] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1d5d8dca, Job ID: ffdaf84c-6330-4260-87cf-ed23f1e668d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:50:02,917-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1d5d8dca] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5355cc17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:50:02,919-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1d5d8dca] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5355cc17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:50:02,930-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [7b9d0be4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:50:02,931-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [7b9d0be4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:54:42,811-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler6) [43207b46] Attempting to update VMs/Templates Ovf.
>2018-05-29 03:54:42,813-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 03:54:42,813-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 03:54:42,814-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:54:42,818-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 03:54:42,822-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 03:54:42,822-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 03:54:42,822-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 03:54:42,822-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 03:54:42,823-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 03:54:42,823-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 03:55:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6d26fb54] Autorecovering 2 hosts
>2018-05-29 03:55:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6d26fb54] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:55:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [675c831] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:55:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [675c831] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:55:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [675c831] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:55:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [675c831] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:55:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [675c831] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4ab778b1
>2018-05-29 03:55:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [675c831] FINISH, SetVdsStatusVDSCommand, log id: 4ab778b1
>2018-05-29 03:55:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [675c831] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:55:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [675c831] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 03:55:00,071-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [675c831] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:55:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2724f5d2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:55:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2724f5d2] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 03:55:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2724f5d2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:55:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2724f5d2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:55:00,084-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2724f5d2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47a2cf98
>2018-05-29 03:55:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2724f5d2] FINISH, SetVdsStatusVDSCommand, log id: 47a2cf98
>2018-05-29 03:55:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [2724f5d2] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6020fcb1
>2018-05-29 03:55:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [2724f5d2] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6020fcb1
>2018-05-29 03:55:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2724f5d2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 03:55:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2724f5d2] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 03:55:00,585-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [4e13cbef] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7e311857
>2018-05-29 03:55:00,587-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [4e13cbef] FINISH, GetHardwareInfoVDSCommand, log id: 7e311857
>2018-05-29 03:55:00,600-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4b2ccd6c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:55:00,608-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [3b40e6b0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:55:00,613-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [78fb4dc5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:55:00,620-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [78fb4dc5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 418a6bc7
>2018-05-29 03:55:00,992-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fb4dc5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:55:00,992-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [78fb4dc5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 418a6bc7
>2018-05-29 03:55:00,996-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fb4dc5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:55:00,996-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [78fb4dc5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:55:01,000-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fb4dc5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:55:01,001-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [78fb4dc5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:55:01,005-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fb4dc5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:55:01,005-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [78fb4dc5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:55:01,009-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [78fb4dc5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:55:01,011-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fb4dc5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 78fb4dc5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:55:01,014-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 31e90e18
>2018-05-29 03:55:01,014-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:55:01,115-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:55:01,115-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:55:01,115-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:55:01,115-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:55:01,115-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 31e90e18
>2018-05-29 03:55:01,115-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:55:01,128-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [7ba5047d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:55:01,130-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7ba5047d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 281ca77
>2018-05-29 03:55:01,132-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7ba5047d] FINISH, SetVdsStatusVDSCommand, log id: 281ca77
>2018-05-29 03:55:01,161-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7ba5047d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7ba5047d, Job ID: 1288c809-d60d-4e9c-8512-0d4058a25b48, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:55:01,166-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7ba5047d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b40e6b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:55:01,169-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7ba5047d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b40e6b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:55:01,180-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [4c9f54a4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 03:55:01,181-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [4c9f54a4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 03:55:01,784-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [e69b249] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 25b0c650
>2018-05-29 03:55:01,786-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [e69b249] FINISH, GetHardwareInfoVDSCommand, log id: 25b0c650
>2018-05-29 03:55:01,798-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [54d8385b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:55:01,808-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [57c512df] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:55:01,813-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [34046212] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 03:55:01,820-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [34046212] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 427ae7b0
>2018-05-29 03:55:02,200-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34046212] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 03:55:02,200-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [34046212] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 427ae7b0
>2018-05-29 03:55:02,204-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34046212] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:55:02,205-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [34046212] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:55:02,208-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34046212] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:55:02,209-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [34046212] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:55:02,213-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34046212] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 03:55:02,214-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [34046212] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 03:55:02,216-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [34046212] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 03:55:02,219-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34046212] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 34046212, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 03:55:02,225-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 44b59bff
>2018-05-29 03:55:02,225-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 03:55:02,325-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:55:02,325-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 03:55:02,325-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 03:55:02,325-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 03:55:02,325-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 44b59bff
>2018-05-29 03:55:02,325-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 03:55:02,338-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [639e10b3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:55:02,339-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [639e10b3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54c1ba12
>2018-05-29 03:55:02,341-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [639e10b3] FINISH, SetVdsStatusVDSCommand, log id: 54c1ba12
>2018-05-29 03:55:02,373-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [639e10b3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 639e10b3, Job ID: d3e779b4-f4ac-43bb-bb1b-b79c14d4be53, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 03:55:02,378-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [639e10b3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 57c512df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 03:55:02,380-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [639e10b3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 57c512df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 03:55:02,391-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [34e74b88] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 03:55:02,392-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [34e74b88] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [f881735] Autorecovering 2 hosts
>2018-05-29 04:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [f881735] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:00:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [602f02a4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:00:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [602f02a4] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [602f02a4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [602f02a4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:00:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [602f02a4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2e30c8ab
>2018-05-29 04:00:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [602f02a4] FINISH, SetVdsStatusVDSCommand, log id: 2e30c8ab
>2018-05-29 04:00:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [602f02a4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:00:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [602f02a4] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:00:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [602f02a4] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:00:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [32f3b05] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:00:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [32f3b05] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:00:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [32f3b05] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:00:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [32f3b05] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:00:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [32f3b05] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4d355584
>2018-05-29 04:00:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [32f3b05] FINISH, SetVdsStatusVDSCommand, log id: 4d355584
>2018-05-29 04:00:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [32f3b05] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3c1cf926
>2018-05-29 04:00:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [32f3b05] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3c1cf926
>2018-05-29 04:00:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [32f3b05] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:00:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [32f3b05] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:00:01,814-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4db7b2e4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4d05189e
>2018-05-29 04:00:01,816-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4db7b2e4] FINISH, GetHardwareInfoVDSCommand, log id: 4d05189e
>2018-05-29 04:00:01,829-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [221eb62a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:00:01,838-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [50216119] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:00:01,843-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [245ad7de] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:00:01,855-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [245ad7de] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: a60e848
>2018-05-29 04:00:02,976-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [41b23382] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 225441b0
>2018-05-29 04:00:02,978-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [41b23382] FINISH, GetHardwareInfoVDSCommand, log id: 225441b0
>2018-05-29 04:00:02,991-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [50f1875d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:00:02,996-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [168480cc] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:00:03,001-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [83d02b0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:00:03,008-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [83d02b0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 75eb3173
>2018-05-29 04:00:03,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [245ad7de] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:00:03,230-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [245ad7de] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: a60e848
>2018-05-29 04:00:03,234-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [245ad7de] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:00:03,235-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [245ad7de] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:00:03,238-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [245ad7de] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:00:03,239-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [245ad7de] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:00:03,243-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [245ad7de] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:00:03,244-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [245ad7de] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:00:03,247-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [245ad7de] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:00:03,250-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [245ad7de] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 245ad7de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:00:03,256-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 22d295b9
>2018-05-29 04:00:03,256-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:00:03,351-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:00:03,351-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:00:03,351-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:00:03,351-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:00:03,351-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 22d295b9
>2018-05-29 04:00:03,351-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:00:03,364-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [39ed2683] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:00:03,366-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [39ed2683] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4ef56c93
>2018-05-29 04:00:03,368-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [39ed2683] FINISH, SetVdsStatusVDSCommand, log id: 4ef56c93
>2018-05-29 04:00:03,384-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [83d02b0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:00:03,384-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [83d02b0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 75eb3173
>2018-05-29 04:00:03,387-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [83d02b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:00:03,388-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [83d02b0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:00:03,392-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [83d02b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:00:03,393-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [83d02b0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:00:03,396-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [83d02b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:00:03,397-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [83d02b0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:00:03,399-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [39ed2683] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 39ed2683, Job ID: bda9ddcf-f491-4d0b-9804-1d4d9553a73c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:00:03,400-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [83d02b0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:00:03,404-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [39ed2683] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 50216119, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:00:03,405-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [83d02b0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 83d02b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:00:03,407-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [39ed2683] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 50216119, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:00:03,408-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2897b611
>2018-05-29 04:00:03,408-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:00:03,419-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [487c0bae] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:00:03,420-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [487c0bae] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:00:03,507-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:00:03,507-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:00:03,507-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:00:03,507-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:00:03,507-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2897b611
>2018-05-29 04:00:03,507-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:00:03,519-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [19442466] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:00:03,521-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [19442466] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1fe5a1ea
>2018-05-29 04:00:03,523-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [19442466] FINISH, SetVdsStatusVDSCommand, log id: 1fe5a1ea
>2018-05-29 04:00:03,528-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [19442466] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 19442466, Job ID: 57e675c5-5737-4488-8284-71494c741110, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:00:03,533-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [19442466] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 168480cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:00:03,535-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [19442466] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 168480cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:00:03,545-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [48f45aef] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:00:03,546-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [48f45aef] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [16f87cdd] Autorecovering 2 hosts
>2018-05-29 04:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [16f87cdd] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:05:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [42c8f9b2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:05:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [42c8f9b2] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:05:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [42c8f9b2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:05:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [42c8f9b2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:05:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [42c8f9b2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61c2aebc
>2018-05-29 04:05:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [42c8f9b2] FINISH, SetVdsStatusVDSCommand, log id: 61c2aebc
>2018-05-29 04:05:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [42c8f9b2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:05:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [42c8f9b2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:05:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [42c8f9b2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:05:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13aeee66] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:05:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13aeee66] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:05:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13aeee66] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:05:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13aeee66] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:05:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [13aeee66] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 77914af1
>2018-05-29 04:05:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [13aeee66] FINISH, SetVdsStatusVDSCommand, log id: 77914af1
>2018-05-29 04:05:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [13aeee66] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5617c7b7
>2018-05-29 04:05:00,104-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [13aeee66] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5617c7b7
>2018-05-29 04:05:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13aeee66] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:05:00,105-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13aeee66] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:05:01,110-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3bc86d71] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4a8a08ee
>2018-05-29 04:05:01,112-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3bc86d71] FINISH, GetHardwareInfoVDSCommand, log id: 4a8a08ee
>2018-05-29 04:05:01,125-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [3bede82c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:05:01,133-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [7151e453] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:05:01,138-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [21227971] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:05:01,143-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [68791671] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 394769a7
>2018-05-29 04:05:01,145-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [68791671] FINISH, GetHardwareInfoVDSCommand, log id: 394769a7
>2018-05-29 04:05:01,151-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [21227971] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: f65bc8c
>2018-05-29 04:05:01,157-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [4e2873f3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:05:01,162-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [71fc162c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:05:01,167-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4a45a80e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:05:01,179-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4a45a80e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 120015de
>2018-05-29 04:05:01,519-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21227971] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:05:01,519-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [21227971] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: f65bc8c
>2018-05-29 04:05:01,522-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21227971] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:05:01,523-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [21227971] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:05:01,527-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21227971] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:05:01,527-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [21227971] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:05:01,531-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21227971] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:05:01,532-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [21227971] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:05:01,534-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [21227971] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:05:01,537-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21227971] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 21227971, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:05:01,543-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 28f822c
>2018-05-29 04:05:01,543-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:05:01,562-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a45a80e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:05:01,562-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4a45a80e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 120015de
>2018-05-29 04:05:01,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a45a80e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:05:01,567-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4a45a80e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:05:01,571-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a45a80e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:05:01,571-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4a45a80e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:05:01,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a45a80e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:05:01,576-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4a45a80e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:05:01,580-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4a45a80e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:05:01,582-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a45a80e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4a45a80e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:05:01,641-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:05:01,641-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:05:01,641-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:05:01,641-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:05:01,641-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 28f822c
>2018-05-29 04:05:01,641-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:05:01,647-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7f9b00a
>2018-05-29 04:05:01,647-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:05:01,654-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [6650a349] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:05:01,655-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6650a349] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3398d046
>2018-05-29 04:05:01,658-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6650a349] FINISH, SetVdsStatusVDSCommand, log id: 3398d046
>2018-05-29 04:05:01,689-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6650a349] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6650a349, Job ID: 76f748a9-7328-497b-8914-bc71bdc7902c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:05:01,694-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6650a349] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7151e453, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:05:01,696-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6650a349] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7151e453, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:05:01,707-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [3bc139b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:05:01,708-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [3bc139b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:05:01,749-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:05:01,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:05:01,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:05:01,749-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:05:01,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7f9b00a
>2018-05-29 04:05:01,749-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:05:01,761-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [52e2cc9a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:05:01,762-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [52e2cc9a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52b4108f
>2018-05-29 04:05:01,764-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [52e2cc9a] FINISH, SetVdsStatusVDSCommand, log id: 52b4108f
>2018-05-29 04:05:01,771-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [52e2cc9a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 52e2cc9a, Job ID: b751dd75-6ae6-46ee-8552-5650f73dcbc1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:05:01,775-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [52e2cc9a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 71fc162c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:05:01,777-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [52e2cc9a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 71fc162c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:05:01,787-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [3c6a20d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:05:01,788-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [3c6a20d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [1a417fe5] Autorecovering 2 hosts
>2018-05-29 04:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [1a417fe5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:10:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [19df4298] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:10:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [19df4298] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:10:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [19df4298] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:10:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [19df4298] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:10:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [19df4298] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6a82ccd5
>2018-05-29 04:10:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [19df4298] FINISH, SetVdsStatusVDSCommand, log id: 6a82ccd5
>2018-05-29 04:10:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [19df4298] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:10:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [19df4298] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:10:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [19df4298] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:10:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [225c88be] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:10:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [225c88be] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:10:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [225c88be] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:10:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [225c88be] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:10:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [225c88be] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d697497
>2018-05-29 04:10:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [225c88be] FINISH, SetVdsStatusVDSCommand, log id: 3d697497
>2018-05-29 04:10:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [225c88be] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 99d5f16
>2018-05-29 04:10:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [225c88be] FINISH, SetHaMaintenanceModeVDSCommand, log id: 99d5f16
>2018-05-29 04:10:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [225c88be] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:10:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [225c88be] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:10:02,686-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [554c4ec6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1f741b1f
>2018-05-29 04:10:02,688-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [554c4ec6] FINISH, GetHardwareInfoVDSCommand, log id: 1f741b1f
>2018-05-29 04:10:02,690-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [18105fa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4312edea
>2018-05-29 04:10:02,692-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [18105fa] FINISH, GetHardwareInfoVDSCommand, log id: 4312edea
>2018-05-29 04:10:02,702-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [5f47d028] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:10:02,706-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [37611b6a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:10:02,711-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4a848800] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:10:02,715-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [697051e4] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:10:02,717-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [24552673] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:10:02,721-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [408834cf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:10:02,723-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [24552673] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 53f119c3
>2018-05-29 04:10:02,728-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [408834cf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7ce7df6d
>2018-05-29 04:10:03,102-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [408834cf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:10:03,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [408834cf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7ce7df6d
>2018-05-29 04:10:03,106-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [408834cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:10:03,107-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [408834cf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:10:03,110-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24552673] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:10:03,110-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [24552673] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 53f119c3
>2018-05-29 04:10:03,111-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [408834cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:10:03,112-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [408834cf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:10:03,114-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24552673] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:10:03,115-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [24552673] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:10:03,116-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [408834cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:10:03,117-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [408834cf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:10:03,119-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24552673] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:10:03,120-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [24552673] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:10:03,120-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [408834cf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:10:03,123-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [408834cf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 408834cf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:10:03,124-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24552673] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:10:03,125-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [24552673] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:10:03,126-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5da98926
>2018-05-29 04:10:03,126-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:10:03,128-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [24552673] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:10:03,130-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24552673] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 24552673, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:10:03,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:10:03,224-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:10:03,224-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:10:03,224-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:10:03,224-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5da98926
>2018-05-29 04:10:03,224-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:10:03,226-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5540c7cc
>2018-05-29 04:10:03,226-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:10:03,238-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [710588c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:10:03,239-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [710588c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53972f97
>2018-05-29 04:10:03,241-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [710588c] FINISH, SetVdsStatusVDSCommand, log id: 53972f97
>2018-05-29 04:10:03,271-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [710588c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 710588c, Job ID: 081febde-a2a1-4461-842b-32afa14d83bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:10:03,275-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [710588c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 697051e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:10:03,278-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [710588c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 697051e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:10:03,288-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [36a9bfde] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:10:03,290-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [36a9bfde] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:10:03,326-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:10:03,326-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:10:03,326-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:10:03,326-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:10:03,326-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5540c7cc
>2018-05-29 04:10:03,326-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:10:03,339-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [423b397c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:10:03,340-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [423b397c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61d22fff
>2018-05-29 04:10:03,342-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [423b397c] FINISH, SetVdsStatusVDSCommand, log id: 61d22fff
>2018-05-29 04:10:03,348-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [423b397c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 423b397c, Job ID: de0e5842-794b-488b-8676-34b304bc5e9e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:10:03,352-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [423b397c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4a848800, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:10:03,354-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [423b397c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4a848800, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:10:03,364-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [1ef23871] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:10:03,365-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [1ef23871] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [18a9448d] Autorecovering 2 hosts
>2018-05-29 04:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [18a9448d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:15:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [426943a5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [426943a5] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [426943a5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [426943a5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:15:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [426943a5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1a9afe45
>2018-05-29 04:15:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [426943a5] FINISH, SetVdsStatusVDSCommand, log id: 1a9afe45
>2018-05-29 04:15:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [426943a5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:15:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [426943a5] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:15:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [426943a5] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:15:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [14b083e5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:15:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [14b083e5] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:15:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [14b083e5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:15:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [14b083e5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:15:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [14b083e5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 35647456
>2018-05-29 04:15:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [14b083e5] FINISH, SetVdsStatusVDSCommand, log id: 35647456
>2018-05-29 04:15:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [14b083e5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 24767e7f
>2018-05-29 04:15:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [14b083e5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 24767e7f
>2018-05-29 04:15:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [14b083e5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:15:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [14b083e5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:15:00,912-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2938fbeb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4e1b2efb
>2018-05-29 04:15:00,913-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2938fbeb] FINISH, GetHardwareInfoVDSCommand, log id: 4e1b2efb
>2018-05-29 04:15:00,914-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [71adde5d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6370f444
>2018-05-29 04:15:00,916-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [71adde5d] FINISH, GetHardwareInfoVDSCommand, log id: 6370f444
>2018-05-29 04:15:00,930-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [619a2101] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:15:00,931-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [2be741a3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:15:00,936-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [51742a47] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:15:00,939-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [7d50309b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:15:00,941-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [62982b89] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:15:00,944-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1e25a5bc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:15:00,957-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1e25a5bc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: ffe5c59
>2018-05-29 04:15:01,001-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [62982b89] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5cfce7e2
>2018-05-29 04:15:01,330-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1e25a5bc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:15:01,330-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1e25a5bc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: ffe5c59
>2018-05-29 04:15:01,333-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1e25a5bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:15:01,334-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1e25a5bc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:15:01,338-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1e25a5bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:15:01,339-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1e25a5bc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:15:01,342-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1e25a5bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:15:01,343-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1e25a5bc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:15:01,346-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1e25a5bc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:15:01,348-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1e25a5bc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1e25a5bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:15:01,356-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 25bb7706
>2018-05-29 04:15:01,356-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:15:01,382-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [62982b89] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:15:01,382-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [62982b89] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5cfce7e2
>2018-05-29 04:15:01,386-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [62982b89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:15:01,387-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [62982b89] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:15:01,391-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [62982b89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:15:01,391-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [62982b89] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:15:01,395-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [62982b89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:15:01,396-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [62982b89] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:15:01,399-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [62982b89] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:15:01,401-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [62982b89] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 62982b89, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:15:01,457-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:15:01,457-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:15:01,457-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:15:01,457-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:15:01,457-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 25bb7706
>2018-05-29 04:15:01,457-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:15:01,459-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5bc905e
>2018-05-29 04:15:01,459-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:15:01,470-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [6cd73f45] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:15:01,471-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6cd73f45] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45b67f98
>2018-05-29 04:15:01,473-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6cd73f45] FINISH, SetVdsStatusVDSCommand, log id: 45b67f98
>2018-05-29 04:15:01,504-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6cd73f45] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6cd73f45, Job ID: e285b416-6c9d-42c6-9d6c-0ab0f8f0be95, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:15:01,509-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6cd73f45] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7d50309b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:15:01,511-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6cd73f45] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7d50309b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:15:01,522-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [535d69af] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:15:01,523-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [535d69af] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:15:01,557-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:15:01,557-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:15:01,557-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:15:01,557-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:15:01,557-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5bc905e
>2018-05-29 04:15:01,557-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:15:01,569-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [7925efeb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:15:01,571-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7925efeb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 183167da
>2018-05-29 04:15:01,573-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7925efeb] FINISH, SetVdsStatusVDSCommand, log id: 183167da
>2018-05-29 04:15:01,578-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7925efeb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7925efeb, Job ID: bf6506f5-18a7-434f-a881-0d1e9cf7dd3f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:15:01,582-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7925efeb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 51742a47, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:15:01,585-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7925efeb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 51742a47, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:15:01,595-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [373fec35] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:15:01,596-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [373fec35] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:20:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7153284b] Autorecovering 2 hosts
>2018-05-29 04:20:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7153284b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:20:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [358b767c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:20:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [358b767c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:20:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [358b767c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:20:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [358b767c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:20:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [358b767c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 55fa9865
>2018-05-29 04:20:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [358b767c] FINISH, SetVdsStatusVDSCommand, log id: 55fa9865
>2018-05-29 04:20:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [358b767c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:20:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [358b767c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:20:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [358b767c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:20:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51e1739f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:20:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51e1739f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:20:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51e1739f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:20:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51e1739f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:20:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [51e1739f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 91bdac2
>2018-05-29 04:20:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [51e1739f] FINISH, SetVdsStatusVDSCommand, log id: 91bdac2
>2018-05-29 04:20:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [51e1739f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 464dcca4
>2018-05-29 04:20:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [51e1739f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 464dcca4
>2018-05-29 04:20:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51e1739f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:20:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51e1739f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:20:02,195-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [35250b49] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 24ee8251
>2018-05-29 04:20:02,197-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [23c1675c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3ecaf582
>2018-05-29 04:20:02,197-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [35250b49] FINISH, GetHardwareInfoVDSCommand, log id: 24ee8251
>2018-05-29 04:20:02,199-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [23c1675c] FINISH, GetHardwareInfoVDSCommand, log id: 3ecaf582
>2018-05-29 04:20:02,211-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [2f4da17b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:20:02,212-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [1a5d91ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:20:02,221-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [35d7c9b2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:20:02,221-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [fcbdfa2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:20:02,226-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5ba0ad23] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:20:02,231-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4f3baffe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:20:02,239-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5ba0ad23] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6a8f7e42
>2018-05-29 04:20:02,244-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4f3baffe] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 569f79b9
>2018-05-29 04:20:02,630-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f3baffe] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:20:02,630-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4f3baffe] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 569f79b9
>2018-05-29 04:20:02,634-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f3baffe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:20:02,635-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4f3baffe] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:20:02,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f3baffe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:20:02,640-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4f3baffe] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:20:02,644-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f3baffe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:20:02,644-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4f3baffe] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:20:02,647-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4f3baffe] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:20:02,650-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f3baffe] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4f3baffe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:20:02,656-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 38d76dbe
>2018-05-29 04:20:02,656-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:20:02,755-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:20:02,755-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:20:02,755-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:20:02,755-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:20:02,755-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 38d76dbe
>2018-05-29 04:20:02,755-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:20:02,769-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [15d0db41] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:20:02,770-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [15d0db41] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4d233c2e
>2018-05-29 04:20:02,772-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [15d0db41] FINISH, SetVdsStatusVDSCommand, log id: 4d233c2e
>2018-05-29 04:20:02,804-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15d0db41] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 15d0db41, Job ID: ec7f5858-2158-48ed-9003-3429aaad816a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:20:02,808-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15d0db41] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: fcbdfa2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:20:02,810-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15d0db41] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: fcbdfa2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:20:02,821-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [606df59b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:20:02,822-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [606df59b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:20:03,613-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ba0ad23] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:20:03,613-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5ba0ad23] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6a8f7e42
>2018-05-29 04:20:03,617-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ba0ad23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:20:03,617-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5ba0ad23] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:20:03,621-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ba0ad23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:20:03,622-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5ba0ad23] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:20:03,625-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ba0ad23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:20:03,626-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5ba0ad23] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:20:03,630-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5ba0ad23] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:20:03,633-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ba0ad23] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5ba0ad23, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:20:03,636-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 57278dfb
>2018-05-29 04:20:03,636-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:20:03,734-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:20:03,734-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:20:03,734-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:20:03,734-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:20:03,734-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 57278dfb
>2018-05-29 04:20:03,734-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:20:03,748-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [6dd16d97] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:20:03,749-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6dd16d97] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5d7f05f0
>2018-05-29 04:20:03,751-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6dd16d97] FINISH, SetVdsStatusVDSCommand, log id: 5d7f05f0
>2018-05-29 04:20:03,756-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6dd16d97] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6dd16d97, Job ID: f2e4e083-b454-4d1c-ae50-b63e04de73ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:20:03,761-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6dd16d97] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 35d7c9b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:20:03,763-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6dd16d97] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 35d7c9b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:20:03,773-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [7f2fe61a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:20:03,774-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [7f2fe61a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [454bbedd] Autorecovering 2 hosts
>2018-05-29 04:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [454bbedd] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:25:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30b8c251] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:25:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30b8c251] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:25:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30b8c251] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:25:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30b8c251] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:25:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [30b8c251] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 134d1805
>2018-05-29 04:25:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [30b8c251] FINISH, SetVdsStatusVDSCommand, log id: 134d1805
>2018-05-29 04:25:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30b8c251] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:25:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30b8c251] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:25:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [30b8c251] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:25:00,136-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78cc1ec9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:25:00,143-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78cc1ec9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:25:00,143-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78cc1ec9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:25:00,143-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78cc1ec9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:25:00,145-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78cc1ec9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 65a88fd6
>2018-05-29 04:25:00,147-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78cc1ec9] FINISH, SetVdsStatusVDSCommand, log id: 65a88fd6
>2018-05-29 04:25:00,150-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [78cc1ec9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5a76e03c
>2018-05-29 04:25:00,152-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [78cc1ec9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5a76e03c
>2018-05-29 04:25:00,152-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78cc1ec9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:25:00,154-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78cc1ec9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:25:00,442-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [119f1299] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2c9dbbd5
>2018-05-29 04:25:00,444-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [119f1299] FINISH, GetHardwareInfoVDSCommand, log id: 2c9dbbd5
>2018-05-29 04:25:00,456-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [216c6169] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:25:00,465-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [6d676a07] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:25:00,470-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [297cda2b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:25:00,476-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [297cda2b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c458299
>2018-05-29 04:25:00,854-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [297cda2b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:25:00,854-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [297cda2b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c458299
>2018-05-29 04:25:00,858-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [297cda2b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:25:00,859-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [297cda2b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:25:00,863-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [297cda2b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:25:00,864-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [297cda2b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:25:00,867-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [297cda2b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:25:00,868-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [297cda2b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:25:00,870-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [297cda2b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:25:00,874-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [297cda2b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 297cda2b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:25:00,876-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 55b0384f
>2018-05-29 04:25:00,876-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:25:00,975-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:25:00,975-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:25:00,975-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:25:00,975-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:25:00,975-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 55b0384f
>2018-05-29 04:25:00,975-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:25:00,989-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [5223efd7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:25:00,990-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5223efd7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2e5e1b36
>2018-05-29 04:25:00,993-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5223efd7] FINISH, SetVdsStatusVDSCommand, log id: 2e5e1b36
>2018-05-29 04:25:01,021-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5223efd7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5223efd7, Job ID: 1057e5a7-68bc-4f0b-b0f9-fda624a346fa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:25:01,026-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5223efd7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6d676a07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:25:01,029-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5223efd7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6d676a07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:25:01,040-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [17f95016] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:25:01,041-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [17f95016] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:25:01,469-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [bf6cc39] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1a585807
>2018-05-29 04:25:01,470-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [bf6cc39] FINISH, GetHardwareInfoVDSCommand, log id: 1a585807
>2018-05-29 04:25:01,483-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [521a6fda] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:25:01,491-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [47a8acec] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:25:01,497-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [aa6cdb4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:25:01,508-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [aa6cdb4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 176d4074
>2018-05-29 04:25:01,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [aa6cdb4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:25:01,882-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [aa6cdb4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 176d4074
>2018-05-29 04:25:01,885-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [aa6cdb4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:25:01,886-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [aa6cdb4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:25:01,890-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [aa6cdb4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:25:01,891-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [aa6cdb4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:25:01,895-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [aa6cdb4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:25:01,895-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [aa6cdb4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:25:01,898-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [aa6cdb4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:25:01,901-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [aa6cdb4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: aa6cdb4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:25:01,908-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 39416bb8
>2018-05-29 04:25:01,908-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:25:02,014-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:25:02,014-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:25:02,015-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:25:02,015-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:25:02,015-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 39416bb8
>2018-05-29 04:25:02,015-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:25:02,029-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [7e33e3e8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:25:02,030-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7e33e3e8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 656ba454
>2018-05-29 04:25:02,032-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7e33e3e8] FINISH, SetVdsStatusVDSCommand, log id: 656ba454
>2018-05-29 04:25:02,064-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7e33e3e8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7e33e3e8, Job ID: 1f2da9c8-2ca0-49ec-b1b8-feb08f492f8c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:25:02,068-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7e33e3e8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 47a8acec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:25:02,071-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7e33e3e8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 47a8acec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:25:02,082-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [4f1401ef] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:25:02,083-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [4f1401ef] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [5229d1] Autorecovering 2 hosts
>2018-05-29 04:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [5229d1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:30:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [18669341] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:30:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [18669341] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:30:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [18669341] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:30:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [18669341] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:30:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [18669341] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13fcc77f
>2018-05-29 04:30:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [18669341] FINISH, SetVdsStatusVDSCommand, log id: 13fcc77f
>2018-05-29 04:30:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [18669341] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:30:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [18669341] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:30:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [18669341] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:30:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3c3b7b90] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:30:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3c3b7b90] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:30:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3c3b7b90] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:30:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3c3b7b90] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:30:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3c3b7b90] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41ece154
>2018-05-29 04:30:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3c3b7b90] FINISH, SetVdsStatusVDSCommand, log id: 41ece154
>2018-05-29 04:30:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [3c3b7b90] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 28dbea73
>2018-05-29 04:30:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [3c3b7b90] FINISH, SetHaMaintenanceModeVDSCommand, log id: 28dbea73
>2018-05-29 04:30:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3c3b7b90] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:30:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3c3b7b90] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:30:01,640-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [23039d7c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5fafdc11
>2018-05-29 04:30:01,642-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [23039d7c] FINISH, GetHardwareInfoVDSCommand, log id: 5fafdc11
>2018-05-29 04:30:01,654-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [1a6fcb0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:30:01,658-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [5a06324] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:30:01,664-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5ece490e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:30:01,720-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5ece490e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1f05ea34
>2018-05-29 04:30:02,104-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ece490e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:30:02,104-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5ece490e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1f05ea34
>2018-05-29 04:30:02,108-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ece490e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:30:02,109-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5ece490e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:30:02,113-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ece490e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:30:02,114-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5ece490e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:30:02,118-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ece490e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:30:02,119-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5ece490e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:30:02,121-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5ece490e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:30:02,124-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ece490e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5ece490e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:30:02,131-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2f02d19e
>2018-05-29 04:30:02,131-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:30:02,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:30:02,228-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:30:02,228-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:30:02,229-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:30:02,229-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2f02d19e
>2018-05-29 04:30:02,229-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:30:02,243-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [68fbaca9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:30:02,244-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [68fbaca9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 245c438e
>2018-05-29 04:30:02,246-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [68fbaca9] FINISH, SetVdsStatusVDSCommand, log id: 245c438e
>2018-05-29 04:30:02,255-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68fbaca9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 68fbaca9, Job ID: ed2e95cc-38fd-4548-b088-d8eb060098dc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:30:02,259-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68fbaca9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5a06324, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:30:02,262-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68fbaca9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5a06324, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:30:02,273-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [e4c2860] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:30:02,274-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [e4c2860] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:30:02,768-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [69dd522c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2c6f396a
>2018-05-29 04:30:02,770-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [69dd522c] FINISH, GetHardwareInfoVDSCommand, log id: 2c6f396a
>2018-05-29 04:30:02,783-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [435f4b96] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:30:02,787-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [3d5ad063] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:30:02,792-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7fe7bf23] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:30:02,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7fe7bf23] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5648e058
>2018-05-29 04:30:03,166-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fe7bf23] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:30:03,166-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7fe7bf23] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5648e058
>2018-05-29 04:30:03,170-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fe7bf23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:30:03,171-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7fe7bf23] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:30:03,175-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fe7bf23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:30:03,175-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7fe7bf23] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:30:03,179-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fe7bf23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:30:03,180-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7fe7bf23] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:30:03,183-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7fe7bf23] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:30:03,185-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fe7bf23] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7fe7bf23, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:30:03,188-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 31fcab85
>2018-05-29 04:30:03,188-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:30:03,285-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:30:03,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:30:03,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:30:03,285-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:30:03,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 31fcab85
>2018-05-29 04:30:03,285-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:30:03,298-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [7e937646] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:30:03,299-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7e937646] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ba1459
>2018-05-29 04:30:03,301-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7e937646] FINISH, SetVdsStatusVDSCommand, log id: 6ba1459
>2018-05-29 04:30:03,306-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7e937646] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7e937646, Job ID: 8d6f7659-2055-4a66-ba48-3a840f7bb0b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:30:03,311-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7e937646] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3d5ad063, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:30:03,313-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7e937646] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3d5ad063, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:30:03,325-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [10f3d4a5] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:30:03,326-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [10f3d4a5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [26439ad0] Autorecovering 2 hosts
>2018-05-29 04:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [26439ad0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:35:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [17d9eb9d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:35:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [17d9eb9d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:35:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [17d9eb9d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:35:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [17d9eb9d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:35:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [17d9eb9d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4c0d7341
>2018-05-29 04:35:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [17d9eb9d] FINISH, SetVdsStatusVDSCommand, log id: 4c0d7341
>2018-05-29 04:35:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [17d9eb9d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:35:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [17d9eb9d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:35:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [17d9eb9d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:35:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [b808a06] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:35:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [b808a06] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:35:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [b808a06] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:35:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [b808a06] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:35:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [b808a06] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6d24bae3
>2018-05-29 04:35:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [b808a06] FINISH, SetVdsStatusVDSCommand, log id: 6d24bae3
>2018-05-29 04:35:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [b808a06] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2879b908
>2018-05-29 04:35:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [b808a06] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2879b908
>2018-05-29 04:35:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [b808a06] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:35:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [b808a06] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:35:01,023-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5d81a297] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6b9b39f3
>2018-05-29 04:35:01,024-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5d81a297] FINISH, GetHardwareInfoVDSCommand, log id: 6b9b39f3
>2018-05-29 04:35:01,037-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [4b4bee7f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:35:01,046-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [5e057866] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:35:01,050-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [63a0e423] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:35:01,057-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [63a0e423] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 78073daa
>2018-05-29 04:35:01,428-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [63a0e423] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:35:01,428-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [63a0e423] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 78073daa
>2018-05-29 04:35:01,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [63a0e423] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:35:01,433-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [63a0e423] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:35:01,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [63a0e423] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:35:01,437-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [63a0e423] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:35:01,441-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [63a0e423] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:35:01,441-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [63a0e423] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:35:01,444-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [63a0e423] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:35:01,447-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [63a0e423] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 63a0e423, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:35:01,450-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5905acae
>2018-05-29 04:35:01,450-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:35:01,548-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:35:01,548-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:35:01,548-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:35:01,548-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:35:01,548-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5905acae
>2018-05-29 04:35:01,548-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:35:01,561-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [7f832f6c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:35:01,562-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7f832f6c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2572afd9
>2018-05-29 04:35:01,564-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7f832f6c] FINISH, SetVdsStatusVDSCommand, log id: 2572afd9
>2018-05-29 04:35:01,593-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f832f6c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7f832f6c, Job ID: 10f74cf0-bb9d-4f71-bf7c-7a77ac6e83ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:35:01,597-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f832f6c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5e057866, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:35:01,599-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f832f6c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5e057866, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:35:01,610-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [f8c7aad] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:35:01,611-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [f8c7aad] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:35:02,993-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3eb26dba] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6f7e063d
>2018-05-29 04:35:02,995-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3eb26dba] FINISH, GetHardwareInfoVDSCommand, log id: 6f7e063d
>2018-05-29 04:35:03,008-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [3e2eec66] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:35:03,012-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6a8bb8dd] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:35:03,017-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5f78d927] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:35:03,024-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5f78d927] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6d1653a0
>2018-05-29 04:35:03,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5f78d927] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:35:03,403-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5f78d927] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6d1653a0
>2018-05-29 04:35:03,407-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5f78d927] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:35:03,408-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5f78d927] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:35:03,412-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5f78d927] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:35:03,413-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5f78d927] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:35:03,417-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5f78d927] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:35:03,418-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5f78d927] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:35:03,421-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5f78d927] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:35:03,423-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5f78d927] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5f78d927, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:35:03,426-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1e972f23
>2018-05-29 04:35:03,426-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:35:03,522-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:35:03,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:35:03,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:35:03,522-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:35:03,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1e972f23
>2018-05-29 04:35:03,522-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:35:03,535-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [799c53d4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:35:03,537-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [799c53d4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7dbf42cb
>2018-05-29 04:35:03,539-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [799c53d4] FINISH, SetVdsStatusVDSCommand, log id: 7dbf42cb
>2018-05-29 04:35:03,544-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [799c53d4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 799c53d4, Job ID: 15ca371e-7dc8-454d-9bea-a21e1b24898b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:35:03,549-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [799c53d4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6a8bb8dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:35:03,551-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [799c53d4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6a8bb8dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:35:03,561-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [3a74569c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:35:03,562-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [3a74569c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [57f55da8] Autorecovering 2 hosts
>2018-05-29 04:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [57f55da8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:40:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70238e63] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70238e63] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70238e63] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70238e63] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:40:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [70238e63] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2a709f70
>2018-05-29 04:40:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [70238e63] FINISH, SetVdsStatusVDSCommand, log id: 2a709f70
>2018-05-29 04:40:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70238e63] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:40:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70238e63] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:40:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [70238e63] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:40:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [675b48d7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [675b48d7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [675b48d7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [675b48d7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:40:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [675b48d7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43aa65b4
>2018-05-29 04:40:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [675b48d7] FINISH, SetVdsStatusVDSCommand, log id: 43aa65b4
>2018-05-29 04:40:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [675b48d7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 389fc14
>2018-05-29 04:40:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [675b48d7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 389fc14
>2018-05-29 04:40:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [675b48d7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:40:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [675b48d7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:40:01,202-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [39d3cdb4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7c361aea
>2018-05-29 04:40:01,204-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [39d3cdb4] FINISH, GetHardwareInfoVDSCommand, log id: 7c361aea
>2018-05-29 04:40:01,217-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [3e65d06d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:40:01,226-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [380fc48d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:40:01,231-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4fec3ad9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:40:01,242-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4fec3ad9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 41c165be
>2018-05-29 04:40:01,630-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4fec3ad9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:40:01,630-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4fec3ad9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 41c165be
>2018-05-29 04:40:01,634-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4fec3ad9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:40:01,635-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4fec3ad9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:40:01,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4fec3ad9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:40:01,639-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4fec3ad9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:40:01,643-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4fec3ad9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:40:01,643-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4fec3ad9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:40:01,690-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4fec3ad9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:40:01,694-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4fec3ad9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4fec3ad9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:40:01,701-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3b21df06
>2018-05-29 04:40:01,701-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:40:01,802-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:40:01,802-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:40:01,802-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:40:01,803-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:40:01,803-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3b21df06
>2018-05-29 04:40:01,803-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:40:01,817-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [47354a18] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:40:01,818-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [47354a18] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 55bdfb88
>2018-05-29 04:40:01,820-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [47354a18] FINISH, SetVdsStatusVDSCommand, log id: 55bdfb88
>2018-05-29 04:40:01,852-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [47354a18] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 47354a18, Job ID: 6c03411b-d7f2-43ca-b0a3-a41806b534cf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:40:01,857-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [47354a18] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 380fc48d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:40:01,859-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [47354a18] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 380fc48d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:40:01,871-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [1c104a1f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:40:01,872-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [1c104a1f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:40:02,389-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [9091bce] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 436b06b1
>2018-05-29 04:40:02,391-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [9091bce] FINISH, GetHardwareInfoVDSCommand, log id: 436b06b1
>2018-05-29 04:40:02,404-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [2b1528a1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:40:02,412-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [1015b461] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:40:02,417-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6eceb310] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:40:02,430-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6eceb310] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 30e6fa96
>2018-05-29 04:40:02,804-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6eceb310] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:40:02,804-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6eceb310] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 30e6fa96
>2018-05-29 04:40:02,808-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6eceb310] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:40:02,809-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6eceb310] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:40:02,812-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6eceb310] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:40:02,813-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6eceb310] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:40:02,817-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6eceb310] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:40:02,818-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6eceb310] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:40:02,820-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6eceb310] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:40:02,823-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6eceb310] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6eceb310, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:40:02,825-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4c0ec71
>2018-05-29 04:40:02,825-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:40:02,922-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:40:02,922-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:40:02,922-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:40:02,922-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:40:02,922-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4c0ec71
>2018-05-29 04:40:02,922-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:40:02,934-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [1287b61e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:40:02,936-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1287b61e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a90e9e4
>2018-05-29 04:40:02,938-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1287b61e] FINISH, SetVdsStatusVDSCommand, log id: 5a90e9e4
>2018-05-29 04:40:02,944-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1287b61e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1287b61e, Job ID: 1f527f82-57af-4c9b-8ec4-195812690328, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:40:02,948-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1287b61e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1015b461, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:40:02,951-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1287b61e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1015b461, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:40:02,962-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [4b795acd] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:40:02,963-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [4b795acd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3ac0a58a] Autorecovering 2 hosts
>2018-05-29 04:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3ac0a58a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:45:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [71ffeccf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:45:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [71ffeccf] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:45:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [71ffeccf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:45:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [71ffeccf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:45:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [71ffeccf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 75a9f62b
>2018-05-29 04:45:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [71ffeccf] FINISH, SetVdsStatusVDSCommand, log id: 75a9f62b
>2018-05-29 04:45:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [71ffeccf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:45:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [71ffeccf] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:45:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [71ffeccf] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:45:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [589666c7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:45:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [589666c7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:45:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [589666c7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:45:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [589666c7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:45:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [589666c7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b37bf77
>2018-05-29 04:45:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [589666c7] FINISH, SetVdsStatusVDSCommand, log id: 1b37bf77
>2018-05-29 04:45:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [589666c7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5c108ac7
>2018-05-29 04:45:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [589666c7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5c108ac7
>2018-05-29 04:45:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [589666c7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:45:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [589666c7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:45:00,636-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6ebe1c89] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4b1108fc
>2018-05-29 04:45:00,639-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6ebe1c89] FINISH, GetHardwareInfoVDSCommand, log id: 4b1108fc
>2018-05-29 04:45:00,651-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [6a7dd307] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:45:00,659-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [6de9922f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:45:00,664-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [7d34dc20] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:45:00,722-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [7d34dc20] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 32479cc6
>2018-05-29 04:45:01,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7d34dc20] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:45:01,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [7d34dc20] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 32479cc6
>2018-05-29 04:45:01,103-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7d34dc20] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:45:01,104-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [7d34dc20] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:45:01,107-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7d34dc20] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:45:01,108-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [7d34dc20] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:45:01,112-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7d34dc20] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:45:01,113-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [7d34dc20] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:45:01,115-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [7d34dc20] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:45:01,118-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7d34dc20] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7d34dc20, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:45:01,124-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 77c434ef
>2018-05-29 04:45:01,124-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:45:01,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:45:01,224-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:45:01,224-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:45:01,224-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:45:01,224-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 77c434ef
>2018-05-29 04:45:01,224-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:45:01,236-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [1e39fa1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:45:01,238-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1e39fa1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 38f89602
>2018-05-29 04:45:01,239-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1e39fa1] FINISH, SetVdsStatusVDSCommand, log id: 38f89602
>2018-05-29 04:45:01,270-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1e39fa1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1e39fa1, Job ID: 9c8524c4-4f8e-426c-b161-47d9c3390f26, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:45:01,275-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1e39fa1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6de9922f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:45:01,277-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1e39fa1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6de9922f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:45:01,287-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [53ca11ff] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:45:01,288-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [53ca11ff] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:45:02,536-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [400aa3da] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2e3feae0
>2018-05-29 04:45:02,537-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [400aa3da] FINISH, GetHardwareInfoVDSCommand, log id: 2e3feae0
>2018-05-29 04:45:02,550-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [26aeb4b2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:45:02,558-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [79ca6b2b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:45:02,563-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [290556f7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:45:02,575-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [290556f7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 21d650b5
>2018-05-29 04:45:02,961-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [290556f7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:45:02,961-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [290556f7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 21d650b5
>2018-05-29 04:45:02,965-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [290556f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:45:02,966-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [290556f7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:45:02,970-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [290556f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:45:02,971-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [290556f7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:45:02,974-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [290556f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:45:02,975-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [290556f7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:45:02,978-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [290556f7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:45:02,981-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [290556f7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 290556f7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:45:02,987-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: f0e1d99
>2018-05-29 04:45:02,987-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:45:03,087-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:45:03,087-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:45:03,087-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:45:03,087-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:45:03,087-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: f0e1d99
>2018-05-29 04:45:03,087-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:45:03,100-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [40e46c43] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:45:03,101-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [40e46c43] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5d0bc469
>2018-05-29 04:45:03,103-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [40e46c43] FINISH, SetVdsStatusVDSCommand, log id: 5d0bc469
>2018-05-29 04:45:03,136-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [40e46c43] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 40e46c43, Job ID: 17bdb476-0199-4852-bc3a-d0aea059b904, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:45:03,141-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [40e46c43] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 79ca6b2b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:45:03,143-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [40e46c43] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 79ca6b2b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:45:03,155-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [309396e6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:45:03,156-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [309396e6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [521dbf19] Autorecovering 2 hosts
>2018-05-29 04:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [521dbf19] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:50:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2c8a5a86] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:50:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2c8a5a86] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:50:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2c8a5a86] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:50:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2c8a5a86] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:50:00,051-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2c8a5a86] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 580beeab
>2018-05-29 04:50:00,054-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2c8a5a86] FINISH, SetVdsStatusVDSCommand, log id: 580beeab
>2018-05-29 04:50:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2c8a5a86] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:50:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2c8a5a86] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:50:00,057-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2c8a5a86] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:50:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3f4c782d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3f4c782d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3f4c782d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3f4c782d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:50:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3f4c782d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f7acc6a
>2018-05-29 04:50:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3f4c782d] FINISH, SetVdsStatusVDSCommand, log id: f7acc6a
>2018-05-29 04:50:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [3f4c782d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7b2f4f10
>2018-05-29 04:50:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [3f4c782d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7b2f4f10
>2018-05-29 04:50:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3f4c782d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:50:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3f4c782d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:50:00,745-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [12540dbf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1acc8e50
>2018-05-29 04:50:00,747-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [12540dbf] FINISH, GetHardwareInfoVDSCommand, log id: 1acc8e50
>2018-05-29 04:50:00,760-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [17aee9c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:50:00,769-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [1735b4ea] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:50:00,774-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [53b70db] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:50:00,781-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [53b70db] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 51557b14
>2018-05-29 04:50:01,151-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [53b70db] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:50:01,151-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [53b70db] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 51557b14
>2018-05-29 04:50:01,155-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [53b70db] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:50:01,156-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [53b70db] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:50:01,159-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [53b70db] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:50:01,160-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [53b70db] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:50:01,164-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [53b70db] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:50:01,165-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [53b70db] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:50:01,169-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [53b70db] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:50:01,171-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [53b70db] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 53b70db, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:50:01,174-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 321f3d6
>2018-05-29 04:50:01,174-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:50:01,270-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:50:01,270-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:50:01,270-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:50:01,270-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:50:01,270-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 321f3d6
>2018-05-29 04:50:01,270-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:50:01,284-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [69aa852c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:50:01,285-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [69aa852c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a43be43
>2018-05-29 04:50:01,288-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [69aa852c] FINISH, SetVdsStatusVDSCommand, log id: a43be43
>2018-05-29 04:50:01,316-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [69aa852c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 69aa852c, Job ID: 6f5b7763-9c5e-450d-a5f9-0e9c59f94078, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:50:01,321-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [69aa852c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1735b4ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:50:01,323-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [69aa852c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1735b4ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:50:01,334-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [52aabb80] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:50:01,335-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [52aabb80] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:50:01,980-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5ffc834f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5cc98090
>2018-05-29 04:50:01,982-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5ffc834f] FINISH, GetHardwareInfoVDSCommand, log id: 5cc98090
>2018-05-29 04:50:01,996-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [c8fbf0c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:50:02,005-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [7fbed239] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:50:02,010-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [27ba0d3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:50:02,022-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [27ba0d3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 380ee207
>2018-05-29 04:50:03,399-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27ba0d3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:50:03,399-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [27ba0d3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 380ee207
>2018-05-29 04:50:03,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27ba0d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:50:03,404-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [27ba0d3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:50:03,408-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27ba0d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:50:03,408-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [27ba0d3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:50:03,412-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27ba0d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:50:03,413-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [27ba0d3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:50:03,417-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [27ba0d3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:50:03,419-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27ba0d3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 27ba0d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:50:03,427-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1ec06fb
>2018-05-29 04:50:03,427-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:50:03,524-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:50:03,524-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:50:03,524-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:50:03,524-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:50:03,524-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1ec06fb
>2018-05-29 04:50:03,524-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:50:03,539-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [542b46fa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:50:03,541-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [542b46fa] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: cd03692
>2018-05-29 04:50:03,543-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [542b46fa] FINISH, SetVdsStatusVDSCommand, log id: cd03692
>2018-05-29 04:50:03,577-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [542b46fa] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 542b46fa, Job ID: 9d312c4a-0329-471c-bba2-23c8dc57e069, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:50:03,582-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [542b46fa] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7fbed239, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:50:03,584-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [542b46fa] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7fbed239, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:50:03,596-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [2b095b34] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:50:03,598-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [2b095b34] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:54:42,824-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler8) [6a6b9572] Attempting to update VMs/Templates Ovf.
>2018-05-29 04:54:42,826-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 04:54:42,826-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 04:54:42,827-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:54:42,876-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 04:54:42,880-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 04:54:42,880-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 04:54:42,880-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 04:54:42,880-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 04:54:42,880-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 04:54:42,881-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 04:55:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [14800dac] Autorecovering 2 hosts
>2018-05-29 04:55:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [14800dac] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:55:00,033-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [51d55153] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:55:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [51d55153] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:55:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [51d55153] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:55:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [51d55153] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:55:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [51d55153] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47299f83
>2018-05-29 04:55:00,045-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [51d55153] FINISH, SetVdsStatusVDSCommand, log id: 47299f83
>2018-05-29 04:55:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [51d55153] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:55:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [51d55153] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 04:55:00,047-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [51d55153] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:55:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cbbfcfd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:55:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cbbfcfd] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 04:55:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cbbfcfd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:55:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cbbfcfd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:55:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2cbbfcfd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54e0eee2
>2018-05-29 04:55:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2cbbfcfd] FINISH, SetVdsStatusVDSCommand, log id: 54e0eee2
>2018-05-29 04:55:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [2cbbfcfd] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5af0f76e
>2018-05-29 04:55:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [2cbbfcfd] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5af0f76e
>2018-05-29 04:55:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cbbfcfd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 04:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cbbfcfd] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 04:55:01,290-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [4f0d183] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2b1091ed
>2018-05-29 04:55:01,291-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [4f0d183] FINISH, GetHardwareInfoVDSCommand, log id: 2b1091ed
>2018-05-29 04:55:01,304-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3f47b872] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:55:01,308-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [2a85646a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:55:01,313-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [713c8fd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:55:01,371-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [713c8fd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7fdc8490
>2018-05-29 04:55:01,750-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [713c8fd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:55:01,750-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [713c8fd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7fdc8490
>2018-05-29 04:55:01,754-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [713c8fd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:55:01,755-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [713c8fd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:55:01,759-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [713c8fd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:55:01,759-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [713c8fd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:55:01,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [713c8fd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:55:01,768-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [713c8fd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:55:01,771-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [713c8fd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:55:01,774-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [713c8fd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 713c8fd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:55:01,781-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 18531e71
>2018-05-29 04:55:01,781-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:55:01,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:55:01,881-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:55:01,881-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:55:01,881-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:55:01,881-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 18531e71
>2018-05-29 04:55:01,881-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:55:01,895-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [7140868f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:55:01,897-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7140868f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 29f58023
>2018-05-29 04:55:01,899-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7140868f] FINISH, SetVdsStatusVDSCommand, log id: 29f58023
>2018-05-29 04:55:01,900-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [487f0b56] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: e38365d
>2018-05-29 04:55:01,902-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [487f0b56] FINISH, GetHardwareInfoVDSCommand, log id: e38365d
>2018-05-29 04:55:01,914-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7140868f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7140868f, Job ID: a32233d3-8fad-45e7-8f51-4cd0f3f5e1c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:55:01,918-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [29c3a564] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:55:01,920-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7140868f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2a85646a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:55:01,923-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7140868f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2a85646a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:55:01,924-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [15aac258] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:55:01,931-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [55ac5bac] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 04:55:01,939-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [499c9fd7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 04:55:01,941-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [499c9fd7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 04:55:01,941-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [55ac5bac] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2f3704b3
>2018-05-29 04:55:02,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55ac5bac] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 04:55:02,327-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [55ac5bac] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2f3704b3
>2018-05-29 04:55:02,331-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55ac5bac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:55:02,332-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [55ac5bac] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:55:02,335-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55ac5bac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:55:02,336-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [55ac5bac] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:55:02,340-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55ac5bac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 04:55:02,340-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [55ac5bac] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 04:55:02,343-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [55ac5bac] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 04:55:02,346-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55ac5bac] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 55ac5bac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 04:55:02,349-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6a93cdaf
>2018-05-29 04:55:02,349-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 04:55:02,448-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:55:02,448-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 04:55:02,448-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 04:55:02,448-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 04:55:02,448-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6a93cdaf
>2018-05-29 04:55:02,448-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 04:55:02,460-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [3af2c9d0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:55:02,462-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3af2c9d0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 522294a7
>2018-05-29 04:55:02,464-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3af2c9d0] FINISH, SetVdsStatusVDSCommand, log id: 522294a7
>2018-05-29 04:55:02,469-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3af2c9d0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3af2c9d0, Job ID: b6ce40cf-06ca-445c-95ad-104699610c9a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 04:55:02,474-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3af2c9d0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 15aac258, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 04:55:02,476-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3af2c9d0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 15aac258, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 04:55:02,487-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [59d1d37f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 04:55:02,488-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [59d1d37f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:00:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [63a320a2] Autorecovering 2 hosts
>2018-05-29 05:00:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [63a320a2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:00:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4c750321] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4c750321] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:00:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4c750321] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:00:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4c750321] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:00:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4c750321] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 203f92fb
>2018-05-29 05:00:00,080-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4c750321] FINISH, SetVdsStatusVDSCommand, log id: 203f92fb
>2018-05-29 05:00:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4c750321] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:00:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4c750321] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:00:00,083-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [4c750321] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:00:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3605089c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3605089c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3605089c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3605089c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:00:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3605089c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 51bce765
>2018-05-29 05:00:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3605089c] FINISH, SetVdsStatusVDSCommand, log id: 51bce765
>2018-05-29 05:00:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [3605089c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 599f5a01
>2018-05-29 05:00:00,104-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [3605089c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 599f5a01
>2018-05-29 05:00:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3605089c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:00:00,105-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3605089c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:00:02,726-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [fc16e08] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 315ed790
>2018-05-29 05:00:02,728-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [fc16e08] FINISH, GetHardwareInfoVDSCommand, log id: 315ed790
>2018-05-29 05:00:02,742-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [25325821] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:00:02,751-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5f7fbfc8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:00:02,756-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [195e613b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:00:02,767-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [195e613b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e91be66
>2018-05-29 05:00:03,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7542ca35] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7e643ae7
>2018-05-29 05:00:03,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7542ca35] FINISH, GetHardwareInfoVDSCommand, log id: 7e643ae7
>2018-05-29 05:00:03,088-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [6bea44e2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:00:03,093-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [6fb48f07] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:00:03,098-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [394a52cc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:00:03,104-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [394a52cc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 492ef51a
>2018-05-29 05:00:03,135-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [195e613b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:00:03,135-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [195e613b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e91be66
>2018-05-29 05:00:03,139-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [195e613b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:00:03,140-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [195e613b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:00:03,143-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [195e613b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:00:03,144-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [195e613b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:00:03,148-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [195e613b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:00:03,148-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [195e613b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:00:03,151-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [195e613b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:00:03,154-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [195e613b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 195e613b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:00:03,160-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2be26b29
>2018-05-29 05:00:03,160-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:00:03,256-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:00:03,256-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:00:03,256-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:00:03,256-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:00:03,256-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2be26b29
>2018-05-29 05:00:03,256-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:00:03,270-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [45e34b3d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:00:03,271-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [45e34b3d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6f1779d1
>2018-05-29 05:00:03,273-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [45e34b3d] FINISH, SetVdsStatusVDSCommand, log id: 6f1779d1
>2018-05-29 05:00:03,305-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [45e34b3d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 45e34b3d, Job ID: 09dae4a3-fbe2-4fe3-9dff-5ef54b753764, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:00:03,309-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [45e34b3d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5f7fbfc8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:00:03,311-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [45e34b3d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5f7fbfc8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:00:03,323-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [591ab818] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:00:03,324-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [591ab818] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:00:03,481-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [394a52cc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:00:03,481-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [394a52cc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 492ef51a
>2018-05-29 05:00:03,485-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [394a52cc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:00:03,486-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [394a52cc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:00:03,490-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [394a52cc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:00:03,491-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [394a52cc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:00:03,494-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [394a52cc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:00:03,495-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [394a52cc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:00:03,498-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [394a52cc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:00:03,500-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [394a52cc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 394a52cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:00:03,503-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2a0ebf03
>2018-05-29 05:00:03,503-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:00:03,602-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:00:03,602-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:00:03,602-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:00:03,602-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:00:03,602-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2a0ebf03
>2018-05-29 05:00:03,602-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:00:03,615-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [4d8d020] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:00:03,616-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4d8d020] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e79574f
>2018-05-29 05:00:03,619-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4d8d020] FINISH, SetVdsStatusVDSCommand, log id: 5e79574f
>2018-05-29 05:00:03,624-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4d8d020] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4d8d020, Job ID: dd3f0010-6ba3-4ece-9649-6b1b2722ee7c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:00:03,628-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4d8d020] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6fb48f07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:00:03,631-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4d8d020] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6fb48f07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:00:03,641-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [634498af] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:00:03,642-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [634498af] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6d8b11d6] Autorecovering 2 hosts
>2018-05-29 05:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6d8b11d6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:05:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [76701550] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:05:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [76701550] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:05:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [76701550] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:05:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [76701550] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:05:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [76701550] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47f67484
>2018-05-29 05:05:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [76701550] FINISH, SetVdsStatusVDSCommand, log id: 47f67484
>2018-05-29 05:05:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [76701550] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:05:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [76701550] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:05:00,073-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [76701550] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:05:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d6fa1a3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:05:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d6fa1a3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:05:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d6fa1a3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:05:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d6fa1a3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:05:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6d6fa1a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26be8864
>2018-05-29 05:05:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6d6fa1a3] FINISH, SetVdsStatusVDSCommand, log id: 26be8864
>2018-05-29 05:05:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6d6fa1a3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 58320987
>2018-05-29 05:05:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6d6fa1a3] FINISH, SetHaMaintenanceModeVDSCommand, log id: 58320987
>2018-05-29 05:05:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d6fa1a3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:05:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d6fa1a3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:05:00,904-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [64f7e9a7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 42121973
>2018-05-29 05:05:00,906-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [64f7e9a7] FINISH, GetHardwareInfoVDSCommand, log id: 42121973
>2018-05-29 05:05:00,919-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [706e4df1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:05:00,928-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [b395b90] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:05:00,933-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [25b214a8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:05:00,940-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [25b214a8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4b42fab1
>2018-05-29 05:05:01,194-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3af8a134] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2a068ef1
>2018-05-29 05:05:01,195-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3af8a134] FINISH, GetHardwareInfoVDSCommand, log id: 2a068ef1
>2018-05-29 05:05:01,208-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [3fd4eb71] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:05:01,212-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [19fd695a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:05:01,217-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [41d25aaa] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:05:01,224-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [41d25aaa] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7cd73fc6
>2018-05-29 05:05:01,304-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [25b214a8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:05:01,304-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [25b214a8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4b42fab1
>2018-05-29 05:05:01,308-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [25b214a8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:05:01,309-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [25b214a8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:05:01,312-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [25b214a8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:05:01,313-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [25b214a8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:05:01,317-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [25b214a8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:05:01,317-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [25b214a8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:05:01,320-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [25b214a8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:05:01,323-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [25b214a8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 25b214a8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:05:01,326-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: f0d96bb
>2018-05-29 05:05:01,326-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:05:01,424-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:05:01,424-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:05:01,424-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:05:01,424-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:05:01,424-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: f0d96bb
>2018-05-29 05:05:01,424-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:05:01,437-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4fd85c0c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:05:01,439-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4fd85c0c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: fb36774
>2018-05-29 05:05:01,441-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4fd85c0c] FINISH, SetVdsStatusVDSCommand, log id: fb36774
>2018-05-29 05:05:01,471-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4fd85c0c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4fd85c0c, Job ID: ac0224b8-9c85-4b71-a4c7-56c0e820f31e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:05:01,476-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4fd85c0c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: b395b90, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:05:01,479-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4fd85c0c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: b395b90, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:05:01,490-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [3b1fb3d0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:05:01,491-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [3b1fb3d0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:05:01,608-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [41d25aaa] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:05:01,608-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [41d25aaa] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7cd73fc6
>2018-05-29 05:05:01,612-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [41d25aaa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:05:01,612-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [41d25aaa] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:05:01,616-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [41d25aaa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:05:01,617-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [41d25aaa] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:05:01,621-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [41d25aaa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:05:01,621-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [41d25aaa] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:05:01,625-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [41d25aaa] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:05:01,628-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [41d25aaa] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 41d25aaa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:05:01,631-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1f239aeb
>2018-05-29 05:05:01,631-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:05:01,732-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:05:01,732-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:05:01,732-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:05:01,732-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:05:01,732-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1f239aeb
>2018-05-29 05:05:01,732-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:05:01,746-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [7633c97] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:05:01,747-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7633c97] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7cb823ec
>2018-05-29 05:05:01,749-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7633c97] FINISH, SetVdsStatusVDSCommand, log id: 7cb823ec
>2018-05-29 05:05:01,755-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7633c97] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7633c97, Job ID: 271bf38d-3f9e-4056-9a69-c7d451e11f9e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:05:01,760-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7633c97] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 19fd695a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:05:01,762-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7633c97] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 19fd695a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:05:01,773-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [5a1336bd] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:05:01,774-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [5a1336bd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [423e1413] Autorecovering 2 hosts
>2018-05-29 05:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [423e1413] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:10:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2a4d302c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2a4d302c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2a4d302c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2a4d302c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:10:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2a4d302c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b55b00f
>2018-05-29 05:10:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2a4d302c] FINISH, SetVdsStatusVDSCommand, log id: 4b55b00f
>2018-05-29 05:10:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2a4d302c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:10:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2a4d302c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:10:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [2a4d302c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:10:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2b600d2c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2b600d2c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2b600d2c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2b600d2c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:10:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2b600d2c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2eae8c60
>2018-05-29 05:10:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2b600d2c] FINISH, SetVdsStatusVDSCommand, log id: 2eae8c60
>2018-05-29 05:10:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [2b600d2c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 41092827
>2018-05-29 05:10:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [2b600d2c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 41092827
>2018-05-29 05:10:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2b600d2c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:10:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2b600d2c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:10:02,395-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3d9f7295] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 47099baf
>2018-05-29 05:10:02,397-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3d9f7295] FINISH, GetHardwareInfoVDSCommand, log id: 47099baf
>2018-05-29 05:10:02,412-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [560c7155] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:10:02,421-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2bc2eb94] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:10:02,426-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2e9a89ed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:10:02,484-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2e9a89ed] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 413b8a8
>2018-05-29 05:10:02,659-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [eaaf1ac] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 665bf7ca
>2018-05-29 05:10:02,660-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [eaaf1ac] FINISH, GetHardwareInfoVDSCommand, log id: 665bf7ca
>2018-05-29 05:10:02,674-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [5cef15ef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:10:02,678-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [399de8b6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:10:02,683-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [16a7b04b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:10:02,690-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [16a7b04b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 30c2c2e3
>2018-05-29 05:10:03,072-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [16a7b04b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:10:03,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [16a7b04b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 30c2c2e3
>2018-05-29 05:10:03,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [16a7b04b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:10:03,077-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [16a7b04b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:10:03,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [16a7b04b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:10:03,082-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [16a7b04b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:10:03,086-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [16a7b04b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:10:03,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [16a7b04b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:10:03,089-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [16a7b04b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:10:03,091-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [16a7b04b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 16a7b04b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:10:03,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 44a989e5
>2018-05-29 05:10:03,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:10:03,197-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:10:03,197-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:10:03,197-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:10:03,197-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:10:03,197-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 44a989e5
>2018-05-29 05:10:03,197-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:10:03,210-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [3a9e58ad] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:10:03,212-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3a9e58ad] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a722bcd
>2018-05-29 05:10:03,214-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3a9e58ad] FINISH, SetVdsStatusVDSCommand, log id: a722bcd
>2018-05-29 05:10:03,246-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3a9e58ad] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3a9e58ad, Job ID: 0dfcefcc-70ec-4cc0-a4d4-0b9f00c5beb8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:10:03,251-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3a9e58ad] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 399de8b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:10:03,253-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3a9e58ad] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 399de8b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:10:03,264-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [2ce9e805] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:10:03,265-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [2ce9e805] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:10:03,850-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e9a89ed] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:10:03,850-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2e9a89ed] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 413b8a8
>2018-05-29 05:10:03,854-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e9a89ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:10:03,855-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2e9a89ed] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:10:03,859-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e9a89ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:10:03,860-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2e9a89ed] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:10:03,864-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e9a89ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:10:03,864-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2e9a89ed] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:10:03,867-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2e9a89ed] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:10:03,870-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e9a89ed] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2e9a89ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:10:03,873-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3c20521f
>2018-05-29 05:10:03,873-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:10:03,968-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:10:03,968-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:10:03,968-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:10:03,968-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:10:03,968-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3c20521f
>2018-05-29 05:10:03,968-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:10:03,981-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [2fa2f736] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:10:03,982-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2fa2f736] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5b3ebca4
>2018-05-29 05:10:03,984-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2fa2f736] FINISH, SetVdsStatusVDSCommand, log id: 5b3ebca4
>2018-05-29 05:10:03,990-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fa2f736] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2fa2f736, Job ID: ce3926e7-b017-40fd-8a52-980fa93282b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:10:03,994-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fa2f736] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2bc2eb94, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:10:03,996-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fa2f736] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2bc2eb94, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:10:04,007-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [2c1b62aa] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:10:04,008-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [2c1b62aa] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:15:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [25412e13] Autorecovering 2 hosts
>2018-05-29 05:15:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [25412e13] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:15:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [950b1e2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:15:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [950b1e2] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:15:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [950b1e2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:15:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [950b1e2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:15:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [950b1e2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7293ea2a
>2018-05-29 05:15:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [950b1e2] FINISH, SetVdsStatusVDSCommand, log id: 7293ea2a
>2018-05-29 05:15:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [950b1e2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:15:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [950b1e2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:15:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [950b1e2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:15:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e9f7f53] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:15:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e9f7f53] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:15:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e9f7f53] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:15:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e9f7f53] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:15:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5e9f7f53] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 27abd083
>2018-05-29 05:15:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5e9f7f53] FINISH, SetVdsStatusVDSCommand, log id: 27abd083
>2018-05-29 05:15:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5e9f7f53] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 599a3009
>2018-05-29 05:15:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5e9f7f53] FINISH, SetHaMaintenanceModeVDSCommand, log id: 599a3009
>2018-05-29 05:15:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e9f7f53] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:15:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e9f7f53] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:15:00,859-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6b8a0475] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1ee5f3f3
>2018-05-29 05:15:00,861-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6b8a0475] FINISH, GetHardwareInfoVDSCommand, log id: 1ee5f3f3
>2018-05-29 05:15:00,874-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7f0b5c33] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:15:00,882-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3b362af5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:15:00,887-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5215ffa8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:15:00,893-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5215ffa8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 65dd5eb0
>2018-05-29 05:15:01,267-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5215ffa8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:15:01,267-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5215ffa8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 65dd5eb0
>2018-05-29 05:15:01,271-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5215ffa8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:15:01,272-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5215ffa8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:15:01,276-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5215ffa8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:15:01,276-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5215ffa8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:15:01,281-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5215ffa8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:15:01,282-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5215ffa8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:15:01,285-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5215ffa8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:15:01,288-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5215ffa8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5215ffa8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:15:01,291-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 36fcb20b
>2018-05-29 05:15:01,291-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:15:01,390-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:15:01,390-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:15:01,390-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:15:01,390-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:15:01,390-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 36fcb20b
>2018-05-29 05:15:01,390-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:15:01,404-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [71b45dcc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:15:01,405-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [71b45dcc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ccf0dd8
>2018-05-29 05:15:01,407-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [71b45dcc] FINISH, SetVdsStatusVDSCommand, log id: 6ccf0dd8
>2018-05-29 05:15:01,436-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [71b45dcc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 71b45dcc, Job ID: 57264229-0e0d-48a5-b80c-4f8421630cd0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:15:01,441-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [71b45dcc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b362af5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:15:01,443-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [71b45dcc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b362af5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:15:01,453-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [2fd51123] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:15:01,454-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [2fd51123] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:15:01,652-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5e03f5f0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 492e0e55
>2018-05-29 05:15:01,654-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5e03f5f0] FINISH, GetHardwareInfoVDSCommand, log id: 492e0e55
>2018-05-29 05:15:01,666-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [4309e266] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:15:01,670-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [7e5a979f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:15:01,676-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5b0548cb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:15:01,682-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5b0548cb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 33dfbf39
>2018-05-29 05:15:02,054-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b0548cb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:15:02,054-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5b0548cb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 33dfbf39
>2018-05-29 05:15:02,058-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b0548cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:15:02,059-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5b0548cb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:15:02,062-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b0548cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:15:02,063-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5b0548cb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:15:02,067-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b0548cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:15:02,067-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5b0548cb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:15:02,070-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5b0548cb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:15:02,073-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b0548cb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5b0548cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:15:02,079-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6d17cd8d
>2018-05-29 05:15:02,079-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:15:02,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:15:02,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:15:02,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:15:02,178-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:15:02,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6d17cd8d
>2018-05-29 05:15:02,178-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:15:02,191-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [aeefeb2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:15:02,193-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [aeefeb2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: bdbd16
>2018-05-29 05:15:02,195-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [aeefeb2] FINISH, SetVdsStatusVDSCommand, log id: bdbd16
>2018-05-29 05:15:02,227-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [aeefeb2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: aeefeb2, Job ID: 6dba3f8f-1f78-457d-90a6-d0178002a89d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:15:02,231-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [aeefeb2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7e5a979f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:15:02,234-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [aeefeb2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7e5a979f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:15:02,244-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [5fc2b209] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:15:02,245-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [5fc2b209] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:20:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4a7a341d] Autorecovering 2 hosts
>2018-05-29 05:20:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4a7a341d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:20:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79bdcacf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:20:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79bdcacf] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:20:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79bdcacf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:20:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79bdcacf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:20:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [79bdcacf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3153f99d
>2018-05-29 05:20:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [79bdcacf] FINISH, SetVdsStatusVDSCommand, log id: 3153f99d
>2018-05-29 05:20:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79bdcacf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:20:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79bdcacf] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:20:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [79bdcacf] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:20:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6ae4e99f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:20:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6ae4e99f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:20:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6ae4e99f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:20:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6ae4e99f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:20:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6ae4e99f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4564d76b
>2018-05-29 05:20:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6ae4e99f] FINISH, SetVdsStatusVDSCommand, log id: 4564d76b
>2018-05-29 05:20:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [6ae4e99f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6d788f63
>2018-05-29 05:20:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [6ae4e99f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6d788f63
>2018-05-29 05:20:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6ae4e99f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:20:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6ae4e99f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:20:02,042-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [271a2d48] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4f36a2d5
>2018-05-29 05:20:02,044-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [271a2d48] FINISH, GetHardwareInfoVDSCommand, log id: 4f36a2d5
>2018-05-29 05:20:02,057-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [4ac3821f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:20:02,062-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [76ad38ce] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:20:02,067-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [daf0038] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:20:02,079-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [daf0038] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 507c3af6
>2018-05-29 05:20:02,459-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [daf0038] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:20:02,459-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [daf0038] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 507c3af6
>2018-05-29 05:20:02,463-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [daf0038] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:20:02,464-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [daf0038] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:20:02,468-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [daf0038] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:20:02,468-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [daf0038] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:20:02,472-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [daf0038] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:20:02,473-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [daf0038] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:20:02,476-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [daf0038] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:20:02,478-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [daf0038] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: daf0038, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:20:02,485-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 18d8a13d
>2018-05-29 05:20:02,485-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:20:02,582-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:20:02,582-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:20:02,582-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:20:02,582-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:20:02,582-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 18d8a13d
>2018-05-29 05:20:02,582-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:20:02,595-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [6c4b9d16] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:20:02,596-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6c4b9d16] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b9f9b1d
>2018-05-29 05:20:02,598-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6c4b9d16] FINISH, SetVdsStatusVDSCommand, log id: 3b9f9b1d
>2018-05-29 05:20:02,607-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6c4b9d16] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6c4b9d16, Job ID: 71f0f28c-f390-49bc-a6dd-af34505425f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:20:02,611-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6c4b9d16] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 76ad38ce, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:20:02,613-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6c4b9d16] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 76ad38ce, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:20:02,624-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [308acb46] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:20:02,625-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [308acb46] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:20:02,875-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5ddc00b3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 598de4c6
>2018-05-29 05:20:02,876-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5ddc00b3] FINISH, GetHardwareInfoVDSCommand, log id: 598de4c6
>2018-05-29 05:20:02,888-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [27032220] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:20:02,893-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [1360f123] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:20:02,898-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2b8c48e4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:20:02,905-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2b8c48e4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 199cf55b
>2018-05-29 05:20:04,219-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2b8c48e4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:20:04,219-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2b8c48e4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 199cf55b
>2018-05-29 05:20:04,223-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2b8c48e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:20:04,224-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2b8c48e4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:20:04,227-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2b8c48e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:20:04,228-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2b8c48e4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:20:04,250-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2b8c48e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:20:04,251-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2b8c48e4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:20:04,255-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2b8c48e4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:20:04,258-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2b8c48e4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2b8c48e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:20:04,261-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 27c2e411
>2018-05-29 05:20:04,261-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:20:04,358-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:20:04,359-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:20:04,359-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:20:04,359-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:20:04,359-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 27c2e411
>2018-05-29 05:20:04,359-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:20:04,372-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [360290d2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:20:04,373-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [360290d2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7908cb2a
>2018-05-29 05:20:04,375-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [360290d2] FINISH, SetVdsStatusVDSCommand, log id: 7908cb2a
>2018-05-29 05:20:04,381-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [360290d2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 360290d2, Job ID: 4daa7888-7be8-4e65-8aec-5c9ef4bef231, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:20:04,385-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [360290d2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1360f123, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:20:04,387-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [360290d2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1360f123, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:20:04,398-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [291b67a3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:20:04,399-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [291b67a3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7d2c2c7a] Autorecovering 2 hosts
>2018-05-29 05:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7d2c2c7a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:25:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4a8de19a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:25:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4a8de19a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:25:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4a8de19a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:25:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4a8de19a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:25:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4a8de19a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f808a3
>2018-05-29 05:25:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4a8de19a] FINISH, SetVdsStatusVDSCommand, log id: 5f808a3
>2018-05-29 05:25:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4a8de19a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:25:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4a8de19a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:25:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4a8de19a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:25:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17019fc5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:25:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17019fc5] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:25:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17019fc5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:25:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17019fc5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:25:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [17019fc5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 606a9d01
>2018-05-29 05:25:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [17019fc5] FINISH, SetVdsStatusVDSCommand, log id: 606a9d01
>2018-05-29 05:25:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [17019fc5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 79357741
>2018-05-29 05:25:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [17019fc5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 79357741
>2018-05-29 05:25:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17019fc5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:25:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17019fc5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:25:00,288-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [79df7166] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 60373059
>2018-05-29 05:25:00,289-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [79df7166] FINISH, GetHardwareInfoVDSCommand, log id: 60373059
>2018-05-29 05:25:00,302-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [51811aff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:25:00,311-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3f03e5e5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:25:00,316-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2c47e59b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:25:00,372-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2c47e59b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f9c59f6
>2018-05-29 05:25:00,757-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c47e59b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:25:00,757-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2c47e59b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6f9c59f6
>2018-05-29 05:25:00,761-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c47e59b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:25:00,761-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2c47e59b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:25:00,765-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c47e59b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:25:00,766-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2c47e59b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:25:00,770-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c47e59b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:25:00,771-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2c47e59b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:25:00,773-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2c47e59b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:25:00,776-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c47e59b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2c47e59b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:25:00,782-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 19759c9e
>2018-05-29 05:25:00,782-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:25:00,882-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:25:00,882-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:25:00,882-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:25:00,882-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:25:00,882-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 19759c9e
>2018-05-29 05:25:00,882-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:25:00,895-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4006b18] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:25:00,897-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4006b18] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f326c7e
>2018-05-29 05:25:00,899-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4006b18] FINISH, SetVdsStatusVDSCommand, log id: 5f326c7e
>2018-05-29 05:25:00,930-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4006b18] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4006b18, Job ID: 8819e555-c06e-4401-b673-b176629854ff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:25:00,935-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4006b18] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3f03e5e5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:25:00,938-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4006b18] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3f03e5e5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:25:00,949-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [30907d38] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:25:00,950-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [30907d38] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:25:02,133-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [12cd66e2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 414b4cbe
>2018-05-29 05:25:02,135-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [12cd66e2] FINISH, GetHardwareInfoVDSCommand, log id: 414b4cbe
>2018-05-29 05:25:02,147-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [3386a231] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:25:02,152-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [4762eace] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:25:02,157-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3a4d45e2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:25:02,163-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3a4d45e2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2ea23563
>2018-05-29 05:25:02,544-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a4d45e2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:25:02,544-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3a4d45e2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2ea23563
>2018-05-29 05:25:02,548-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a4d45e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:25:02,549-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3a4d45e2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:25:02,553-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a4d45e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:25:02,554-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3a4d45e2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:25:02,558-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a4d45e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:25:02,558-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3a4d45e2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:25:02,561-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3a4d45e2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:25:02,563-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a4d45e2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3a4d45e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:25:02,566-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 52bfc385
>2018-05-29 05:25:02,566-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:25:02,666-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:25:02,666-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:25:02,666-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:25:02,666-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:25:02,666-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 52bfc385
>2018-05-29 05:25:02,666-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:25:02,679-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4e10d21e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:25:02,680-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4e10d21e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 21b5bfdd
>2018-05-29 05:25:02,683-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4e10d21e] FINISH, SetVdsStatusVDSCommand, log id: 21b5bfdd
>2018-05-29 05:25:02,688-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4e10d21e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4e10d21e, Job ID: 2aed3453-b0f6-4ad6-8319-903517acf2a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:25:02,692-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4e10d21e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4762eace, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:25:02,695-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4e10d21e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4762eace, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:25:02,705-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [5c63ed7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:25:02,706-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [5c63ed7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:30:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [745ce051] Autorecovering 2 hosts
>2018-05-29 05:30:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [745ce051] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:30:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [49d07df] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:30:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [49d07df] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:30:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [49d07df] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:30:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [49d07df] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:30:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [49d07df] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2b96d559
>2018-05-29 05:30:00,081-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [49d07df] FINISH, SetVdsStatusVDSCommand, log id: 2b96d559
>2018-05-29 05:30:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [49d07df] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:30:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [49d07df] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:30:00,084-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [49d07df] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:30:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [312ff299] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:30:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [312ff299] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:30:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [312ff299] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:30:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [312ff299] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:30:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [312ff299] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 777630f0
>2018-05-29 05:30:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [312ff299] FINISH, SetVdsStatusVDSCommand, log id: 777630f0
>2018-05-29 05:30:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [312ff299] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 55673cdc
>2018-05-29 05:30:00,104-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [312ff299] FINISH, SetHaMaintenanceModeVDSCommand, log id: 55673cdc
>2018-05-29 05:30:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [312ff299] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:30:00,108-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [312ff299] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 312ff299, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-29 05:30:00,108-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [312ff299] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:30:00,368-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7d5ba950] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 44d9afbe
>2018-05-29 05:30:00,369-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7d5ba950] FINISH, GetHardwareInfoVDSCommand, log id: 44d9afbe
>2018-05-29 05:30:00,382-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [2c68d0aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:30:00,390-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [20c8d7af] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:30:00,396-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [45cb8c5f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:30:00,407-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [45cb8c5f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3f4b1cd2
>2018-05-29 05:30:00,788-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [45cb8c5f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:30:00,788-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [45cb8c5f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3f4b1cd2
>2018-05-29 05:30:00,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [45cb8c5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:30:00,793-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [45cb8c5f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:30:00,796-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [45cb8c5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:30:00,797-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [45cb8c5f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:30:00,801-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [45cb8c5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:30:00,801-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [45cb8c5f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:30:00,804-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [45cb8c5f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:30:00,807-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [45cb8c5f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 45cb8c5f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:30:00,813-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 37dfaec6
>2018-05-29 05:30:00,813-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:30:00,926-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:30:00,926-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:30:00,926-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:30:00,927-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:30:00,927-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 37dfaec6
>2018-05-29 05:30:00,927-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:30:00,940-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [78f178cc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:30:00,941-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [78f178cc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 225bdd0e
>2018-05-29 05:30:00,943-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [78f178cc] FINISH, SetVdsStatusVDSCommand, log id: 225bdd0e
>2018-05-29 05:30:00,977-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [78f178cc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 78f178cc, Job ID: 9fefc0fe-f3dc-4e63-9c57-9ccc6780c679, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:30:00,981-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [78f178cc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 20c8d7af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:30:00,984-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [78f178cc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 20c8d7af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:30:00,995-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [3a7607c2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:30:00,996-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [3a7607c2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:30:01,529-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1ae57bec] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 791f84be
>2018-05-29 05:30:01,531-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1ae57bec] FINISH, GetHardwareInfoVDSCommand, log id: 791f84be
>2018-05-29 05:30:01,543-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [5ee58503] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:30:01,547-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [6d623fb8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:30:01,552-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [1ced95e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:30:01,559-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [1ced95e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1f54bfc6
>2018-05-29 05:30:01,936-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1ced95e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:30:01,936-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [1ced95e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1f54bfc6
>2018-05-29 05:30:01,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1ced95e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:30:01,940-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [1ced95e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:30:01,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1ced95e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:30:01,945-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [1ced95e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:30:01,949-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1ced95e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:30:01,950-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [1ced95e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:30:01,952-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [1ced95e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:30:01,955-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1ced95e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1ced95e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:30:01,958-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 237f2db2
>2018-05-29 05:30:01,958-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:30:02,057-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:30:02,057-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:30:02,057-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:30:02,057-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:30:02,057-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 237f2db2
>2018-05-29 05:30:02,057-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:30:02,070-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [56e0d89b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:30:02,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [56e0d89b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5ce9bcd6
>2018-05-29 05:30:02,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [56e0d89b] FINISH, SetVdsStatusVDSCommand, log id: 5ce9bcd6
>2018-05-29 05:30:02,108-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [56e0d89b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 56e0d89b, Job ID: 60832dfb-5d40-4667-ac0d-7b12fe3dde56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:30:02,112-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [56e0d89b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6d623fb8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:30:02,114-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [56e0d89b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6d623fb8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:30:02,125-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [1ab38b2d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:30:02,126-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [1ab38b2d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:35:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [758ebd08] Autorecovering 2 hosts
>2018-05-29 05:35:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [758ebd08] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:35:00,032-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5f44b03e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:35:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5f44b03e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:35:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5f44b03e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:35:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5f44b03e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:35:00,040-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5f44b03e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3a7eb3b3
>2018-05-29 05:35:00,043-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5f44b03e] FINISH, SetVdsStatusVDSCommand, log id: 3a7eb3b3
>2018-05-29 05:35:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5f44b03e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:35:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5f44b03e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:35:00,046-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [5f44b03e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:35:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [180f10da] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:35:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [180f10da] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:35:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [180f10da] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:35:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [180f10da] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:35:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [180f10da] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5582838d
>2018-05-29 05:35:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [180f10da] FINISH, SetVdsStatusVDSCommand, log id: 5582838d
>2018-05-29 05:35:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [180f10da] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1b91382
>2018-05-29 05:35:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [180f10da] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1b91382
>2018-05-29 05:35:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [180f10da] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:35:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [180f10da] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:35:01,660-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [229a9879] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2b3908ec
>2018-05-29 05:35:01,662-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [229a9879] FINISH, GetHardwareInfoVDSCommand, log id: 2b3908ec
>2018-05-29 05:35:01,675-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [757cbfcf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:35:01,679-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6bfe2a69] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:35:01,684-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [de56bd0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:35:01,743-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [de56bd0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f8c4e2e
>2018-05-29 05:35:02,124-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [de56bd0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:35:02,124-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [de56bd0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f8c4e2e
>2018-05-29 05:35:02,128-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [de56bd0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:35:02,129-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [de56bd0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:35:02,132-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [de56bd0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:35:02,133-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [de56bd0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:35:02,137-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [de56bd0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:35:02,138-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [de56bd0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:35:02,141-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [de56bd0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:35:02,144-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [de56bd0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: de56bd0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:35:02,150-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6275ec0e
>2018-05-29 05:35:02,150-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:35:02,248-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:35:02,248-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:35:02,248-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:35:02,248-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:35:02,248-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6275ec0e
>2018-05-29 05:35:02,248-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:35:02,261-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [61f761d7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:35:02,263-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [61f761d7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 62ae9350
>2018-05-29 05:35:02,265-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [61f761d7] FINISH, SetVdsStatusVDSCommand, log id: 62ae9350
>2018-05-29 05:35:02,298-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [61f761d7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 61f761d7, Job ID: 5305a0be-0f0a-4f0e-baf8-dc7749c82f4c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:35:02,302-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [61f761d7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6bfe2a69, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:35:02,304-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [61f761d7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6bfe2a69, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:35:02,316-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [7aa5c841] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:35:02,317-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [7aa5c841] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:35:02,698-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [59053644] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1ea56128
>2018-05-29 05:35:02,700-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [59053644] FINISH, GetHardwareInfoVDSCommand, log id: 1ea56128
>2018-05-29 05:35:02,712-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [6c7c22fe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:35:02,717-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [44f7fd3e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:35:02,722-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [334a6c83] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:35:02,734-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [334a6c83] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1badb7ac
>2018-05-29 05:35:03,108-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [334a6c83] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:35:03,108-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [334a6c83] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1badb7ac
>2018-05-29 05:35:03,112-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [334a6c83] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:35:03,113-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [334a6c83] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:35:03,117-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [334a6c83] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:35:03,118-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [334a6c83] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:35:03,122-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [334a6c83] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:35:03,122-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [334a6c83] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:35:03,125-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [334a6c83] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:35:03,128-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [334a6c83] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 334a6c83, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:35:03,130-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 66fa9acd
>2018-05-29 05:35:03,130-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:35:03,227-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:35:03,227-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:35:03,227-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:35:03,227-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:35:03,227-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 66fa9acd
>2018-05-29 05:35:03,227-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:35:03,240-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [e540ccc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:35:03,242-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [e540ccc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 63a9b7ba
>2018-05-29 05:35:03,244-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [e540ccc] FINISH, SetVdsStatusVDSCommand, log id: 63a9b7ba
>2018-05-29 05:35:03,249-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [e540ccc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: e540ccc, Job ID: cd07f803-50e4-4596-b8cb-61bf9885dc6d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:35:03,254-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [e540ccc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 44f7fd3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:35:03,256-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [e540ccc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 44f7fd3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:35:03,267-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [28e24d49] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:35:03,268-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [28e24d49] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:40:00,007-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6006be9b] Autorecovering 2 hosts
>2018-05-29 05:40:00,007-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6006be9b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:40:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [45d0c99f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:40:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [45d0c99f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:40:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [45d0c99f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:40:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [45d0c99f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:40:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [45d0c99f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 34efdba8
>2018-05-29 05:40:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [45d0c99f] FINISH, SetVdsStatusVDSCommand, log id: 34efdba8
>2018-05-29 05:40:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [45d0c99f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:40:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [45d0c99f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:40:00,073-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [45d0c99f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:40:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [129cf305] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:40:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [129cf305] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:40:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [129cf305] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:40:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [129cf305] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:40:00,086-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [129cf305] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25fcdced
>2018-05-29 05:40:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [129cf305] FINISH, SetVdsStatusVDSCommand, log id: 25fcdced
>2018-05-29 05:40:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [129cf305] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6b794bc9
>2018-05-29 05:40:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [129cf305] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6b794bc9
>2018-05-29 05:40:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [129cf305] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:40:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [129cf305] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:40:00,890-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6e60e952] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7c23fb68
>2018-05-29 05:40:00,891-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6e60e952] FINISH, GetHardwareInfoVDSCommand, log id: 7c23fb68
>2018-05-29 05:40:00,904-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [6f01d241] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:40:00,912-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [48868cfa] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:40:00,918-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [38462093] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:40:00,929-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [38462093] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3b10ff3b
>2018-05-29 05:40:01,307-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38462093] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:40:01,307-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [38462093] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3b10ff3b
>2018-05-29 05:40:01,311-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38462093] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:40:01,312-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [38462093] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:40:01,316-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38462093] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:40:01,316-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [38462093] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:40:01,320-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38462093] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:40:01,321-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [38462093] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:40:01,323-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [38462093] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:40:01,326-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38462093] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 38462093, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:40:01,333-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 75855d11
>2018-05-29 05:40:01,333-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:40:01,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:40:01,432-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:40:01,432-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:40:01,432-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:40:01,432-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 75855d11
>2018-05-29 05:40:01,432-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:40:01,444-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [2f6dfb50] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:40:01,445-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2f6dfb50] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2f633746
>2018-05-29 05:40:01,447-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2f6dfb50] FINISH, SetVdsStatusVDSCommand, log id: 2f633746
>2018-05-29 05:40:01,479-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f6dfb50] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2f6dfb50, Job ID: 9ef40f70-b9dc-42d5-9ec7-278cb9df25cf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:40:01,483-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f6dfb50] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 48868cfa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:40:01,486-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f6dfb50] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 48868cfa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:40:01,496-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [3c0d7061] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:40:01,497-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [3c0d7061] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:40:03,079-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2885a07b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 372a78c7
>2018-05-29 05:40:03,081-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2885a07b] FINISH, GetHardwareInfoVDSCommand, log id: 372a78c7
>2018-05-29 05:40:03,094-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [3705db16] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:40:03,103-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [5ea2bece] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:40:03,108-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51dd2abf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:40:03,114-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51dd2abf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7bc90c67
>2018-05-29 05:40:03,510-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dd2abf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:40:03,510-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51dd2abf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7bc90c67
>2018-05-29 05:40:03,514-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dd2abf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:40:03,515-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51dd2abf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:40:03,519-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dd2abf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:40:03,519-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51dd2abf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:40:03,523-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dd2abf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:40:03,524-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51dd2abf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:40:03,526-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51dd2abf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:40:03,529-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dd2abf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 51dd2abf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:40:03,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 62619972
>2018-05-29 05:40:03,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:40:03,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:40:03,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:40:03,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:40:03,627-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:40:03,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 62619972
>2018-05-29 05:40:03,627-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:40:03,639-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [46580a2c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:40:03,640-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [46580a2c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 514616ed
>2018-05-29 05:40:03,642-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [46580a2c] FINISH, SetVdsStatusVDSCommand, log id: 514616ed
>2018-05-29 05:40:03,673-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [46580a2c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 46580a2c, Job ID: 7fb5e6c7-c5df-49e7-a271-7bbda5526baf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:40:03,678-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [46580a2c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5ea2bece, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:40:03,680-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [46580a2c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5ea2bece, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:40:03,690-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [51980f46] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:40:03,691-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [51980f46] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [30da610d] Autorecovering 2 hosts
>2018-05-29 05:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [30da610d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:45:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52f83e5b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52f83e5b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52f83e5b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52f83e5b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:45:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [52f83e5b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7eaa25fe
>2018-05-29 05:45:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [52f83e5b] FINISH, SetVdsStatusVDSCommand, log id: 7eaa25fe
>2018-05-29 05:45:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52f83e5b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:45:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52f83e5b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:45:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [52f83e5b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:45:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5aaab1e1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:45:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5aaab1e1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:45:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5aaab1e1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:45:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5aaab1e1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:45:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5aaab1e1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72a48cbf
>2018-05-29 05:45:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5aaab1e1] FINISH, SetVdsStatusVDSCommand, log id: 72a48cbf
>2018-05-29 05:45:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [5aaab1e1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 31990db1
>2018-05-29 05:45:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [5aaab1e1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 31990db1
>2018-05-29 05:45:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5aaab1e1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:45:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5aaab1e1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:45:01,352-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [c8f7cda] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2e3c7c2f
>2018-05-29 05:45:01,354-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [c8f7cda] FINISH, GetHardwareInfoVDSCommand, log id: 2e3c7c2f
>2018-05-29 05:45:01,367-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [5e134719] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:45:01,376-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [1dfcaa1c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:45:01,382-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7b673956] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:45:01,393-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7b673956] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1617786
>2018-05-29 05:45:01,761-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7b673956] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:45:01,761-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7b673956] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1617786
>2018-05-29 05:45:01,765-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7b673956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:45:01,765-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7b673956] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:45:01,769-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7b673956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:45:01,770-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7b673956] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:45:01,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7b673956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:45:01,774-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7b673956] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:45:01,777-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7b673956] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:45:01,779-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7b673956] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7b673956, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:45:01,786-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2c404fa4
>2018-05-29 05:45:01,786-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:45:01,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:45:01,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:45:01,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:45:01,886-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:45:01,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2c404fa4
>2018-05-29 05:45:01,886-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:45:01,899-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [4eeeadc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:45:01,900-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4eeeadc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2e887ed8
>2018-05-29 05:45:01,902-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4eeeadc] FINISH, SetVdsStatusVDSCommand, log id: 2e887ed8
>2018-05-29 05:45:01,936-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4eeeadc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4eeeadc, Job ID: 0d177def-7701-4b0a-aede-b1746ef079d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:45:01,940-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4eeeadc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1dfcaa1c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:45:01,942-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4eeeadc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1dfcaa1c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:45:01,954-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [58c43eea] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:45:01,955-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [58c43eea] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:45:02,034-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [486fc4ed] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3e2f2e2b
>2018-05-29 05:45:02,036-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [486fc4ed] FINISH, GetHardwareInfoVDSCommand, log id: 3e2f2e2b
>2018-05-29 05:45:02,049-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [774db8c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:45:02,053-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1148001e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:45:02,059-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [67d1746e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:45:02,071-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [67d1746e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 14b39cb4
>2018-05-29 05:45:02,459-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [67d1746e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:45:02,459-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [67d1746e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 14b39cb4
>2018-05-29 05:45:02,463-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [67d1746e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:45:02,464-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [67d1746e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:45:02,467-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [67d1746e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:45:02,468-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [67d1746e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:45:02,472-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [67d1746e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:45:02,473-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [67d1746e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:45:02,476-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [67d1746e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:45:02,478-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [67d1746e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 67d1746e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:45:02,485-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6c1e6eba
>2018-05-29 05:45:02,485-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:45:02,583-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:45:02,583-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:45:02,583-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:45:02,583-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:45:02,583-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6c1e6eba
>2018-05-29 05:45:02,583-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:45:02,596-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [6e6f5978] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:45:02,598-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6e6f5978] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 68f18f45
>2018-05-29 05:45:02,600-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6e6f5978] FINISH, SetVdsStatusVDSCommand, log id: 68f18f45
>2018-05-29 05:45:02,609-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6e6f5978] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6e6f5978, Job ID: 82028b81-d238-46ac-9f45-d2ed843ab5ee, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:45:02,613-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6e6f5978] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1148001e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:45:02,615-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6e6f5978] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1148001e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:45:02,627-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [53cb9cd] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:45:02,628-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [53cb9cd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5831c840] Autorecovering 2 hosts
>2018-05-29 05:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5831c840] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:50:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5350da13] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:50:00,118-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5350da13] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:50:00,118-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5350da13] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:50:00,118-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5350da13] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:50:00,120-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5350da13] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4ae1310e
>2018-05-29 05:50:00,123-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5350da13] FINISH, SetVdsStatusVDSCommand, log id: 4ae1310e
>2018-05-29 05:50:00,124-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5350da13] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:50:00,128-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5350da13] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 5350da13, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-29 05:50:00,128-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5350da13] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:50:00,128-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5350da13] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:50:00,135-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a4ccc14] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:50:00,141-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a4ccc14] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:50:00,141-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a4ccc14] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:50:00,141-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a4ccc14] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:50:00,142-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2a4ccc14] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18841170
>2018-05-29 05:50:00,145-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2a4ccc14] FINISH, SetVdsStatusVDSCommand, log id: 18841170
>2018-05-29 05:50:00,147-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2a4ccc14] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 397bae1a
>2018-05-29 05:50:00,149-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2a4ccc14] FINISH, SetHaMaintenanceModeVDSCommand, log id: 397bae1a
>2018-05-29 05:50:00,149-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a4ccc14] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:50:00,151-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a4ccc14] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:50:03,038-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [1220b34e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3d4bc298
>2018-05-29 05:50:03,040-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [1220b34e] FINISH, GetHardwareInfoVDSCommand, log id: 3d4bc298
>2018-05-29 05:50:03,054-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [75c43e4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:50:03,063-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [7f8d42d1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:50:03,070-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [18298682] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:50:03,129-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [18298682] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2bbbcdfc
>2018-05-29 05:50:03,190-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [340320a5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6466a0bd
>2018-05-29 05:50:03,192-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [340320a5] FINISH, GetHardwareInfoVDSCommand, log id: 6466a0bd
>2018-05-29 05:50:03,205-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4793517] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:50:03,210-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [e731268] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:50:03,215-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6e27f12d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:50:03,221-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6e27f12d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c06a3f
>2018-05-29 05:50:03,512-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [18298682] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:50:03,512-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [18298682] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2bbbcdfc
>2018-05-29 05:50:03,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [18298682] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:50:03,517-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [18298682] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:50:03,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [18298682] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:50:03,522-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [18298682] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:50:03,526-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [18298682] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:50:03,527-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [18298682] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:50:03,530-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [18298682] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:50:03,532-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [18298682] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 18298682, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:50:03,539-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2038c81f
>2018-05-29 05:50:03,539-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:50:03,600-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6e27f12d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:50:03,600-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6e27f12d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c06a3f
>2018-05-29 05:50:03,604-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6e27f12d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:50:03,604-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6e27f12d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:50:03,608-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6e27f12d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:50:03,609-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6e27f12d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:50:03,613-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6e27f12d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:50:03,614-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6e27f12d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:50:03,616-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6e27f12d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:50:03,619-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6e27f12d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6e27f12d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:50:03,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:50:03,639-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:50:03,639-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:50:03,639-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:50:03,639-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2038c81f
>2018-05-29 05:50:03,639-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:50:03,641-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2a88901
>2018-05-29 05:50:03,641-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:50:03,653-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [133086f7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:50:03,654-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [133086f7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: e56b1a8
>2018-05-29 05:50:03,656-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [133086f7] FINISH, SetVdsStatusVDSCommand, log id: e56b1a8
>2018-05-29 05:50:03,689-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [133086f7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 133086f7, Job ID: 84a5b2c6-02c5-443b-9068-479573be8414, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:50:03,693-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [133086f7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7f8d42d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:50:03,696-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [133086f7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7f8d42d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:50:03,706-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [b72a530] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:50:03,707-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [b72a530] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:50:03,739-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:50:03,739-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:50:03,739-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:50:03,739-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:50:03,739-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2a88901
>2018-05-29 05:50:03,739-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:50:03,752-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [8149543] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:50:03,753-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [8149543] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6213560e
>2018-05-29 05:50:03,756-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [8149543] FINISH, SetVdsStatusVDSCommand, log id: 6213560e
>2018-05-29 05:50:03,761-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8149543] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 8149543, Job ID: 0bc2224b-ee2e-4247-a832-36ddac89f0bd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:50:03,765-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8149543] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: e731268, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:50:03,767-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8149543] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: e731268, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:50:03,778-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [3de7bc87] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:50:03,779-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [3de7bc87] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:54:42,881-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler6) [54e52a9a] Attempting to update VMs/Templates Ovf.
>2018-05-29 05:54:42,884-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 05:54:42,884-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 05:54:42,885-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:54:42,889-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 05:54:42,892-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 05:54:42,892-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 05:54:42,892-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 05:54:42,892-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 05:54:42,893-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 05:54:42,893-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 05:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1334ce39] Autorecovering 2 hosts
>2018-05-29 05:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1334ce39] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:55:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1a43e11d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:55:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1a43e11d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:55:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1a43e11d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:55:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1a43e11d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:55:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1a43e11d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25804489
>2018-05-29 05:55:00,080-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1a43e11d] FINISH, SetVdsStatusVDSCommand, log id: 25804489
>2018-05-29 05:55:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1a43e11d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:55:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1a43e11d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 05:55:00,084-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1a43e11d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:55:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [697e2e66] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:55:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [697e2e66] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 05:55:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [697e2e66] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:55:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [697e2e66] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:55:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [697e2e66] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30d77e22
>2018-05-29 05:55:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [697e2e66] FINISH, SetVdsStatusVDSCommand, log id: 30d77e22
>2018-05-29 05:55:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [697e2e66] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 71429938
>2018-05-29 05:55:00,103-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [697e2e66] FINISH, SetHaMaintenanceModeVDSCommand, log id: 71429938
>2018-05-29 05:55:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [697e2e66] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 05:55:00,105-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [697e2e66] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 05:55:01,325-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2e6ba596] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5d80e857
>2018-05-29 05:55:01,327-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2e6ba596] FINISH, GetHardwareInfoVDSCommand, log id: 5d80e857
>2018-05-29 05:55:01,330-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3f449f2d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 68568fb5
>2018-05-29 05:55:01,332-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3f449f2d] FINISH, GetHardwareInfoVDSCommand, log id: 68568fb5
>2018-05-29 05:55:01,342-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [13acbf89] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:55:01,346-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [22049bcc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:55:01,351-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [50492d31] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:55:01,356-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [1d084fbe] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:55:01,357-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [70fe7a31] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:55:01,362-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [f03cae6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 05:55:01,370-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [70fe7a31] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 16697e99
>2018-05-29 05:55:01,374-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [f03cae6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 39a8d136
>2018-05-29 05:55:01,747-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [70fe7a31] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:55:01,748-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [70fe7a31] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 16697e99
>2018-05-29 05:55:01,752-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [70fe7a31] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:55:01,752-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [70fe7a31] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:55:01,756-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [70fe7a31] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:55:01,757-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [70fe7a31] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:55:01,761-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [70fe7a31] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:55:01,761-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [70fe7a31] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:55:01,764-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [70fe7a31] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:55:01,767-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [70fe7a31] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 70fe7a31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:55:01,773-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 63264b5b
>2018-05-29 05:55:01,773-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:55:01,871-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:55:01,871-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:55:01,872-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:55:01,872-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:55:01,872-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 63264b5b
>2018-05-29 05:55:01,872-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:55:01,885-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [7597c16c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:55:01,886-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7597c16c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7913379a
>2018-05-29 05:55:01,888-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7597c16c] FINISH, SetVdsStatusVDSCommand, log id: 7913379a
>2018-05-29 05:55:01,921-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7597c16c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7597c16c, Job ID: b847008d-a016-494d-9b02-e41b4c690a14, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:55:01,926-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7597c16c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 50492d31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:55:01,928-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7597c16c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 50492d31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:55:01,940-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [5ccf1578] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 05:55:01,941-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [5ccf1578] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 05:55:02,743-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [f03cae6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 05:55:02,743-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [f03cae6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 39a8d136
>2018-05-29 05:55:02,746-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [f03cae6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:55:02,747-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [f03cae6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:55:02,751-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [f03cae6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:55:02,752-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [f03cae6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:55:02,756-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [f03cae6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 05:55:02,757-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [f03cae6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 05:55:02,761-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [f03cae6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 05:55:02,763-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [f03cae6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: f03cae6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 05:55:02,766-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 603c2b3e
>2018-05-29 05:55:02,766-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 05:55:02,869-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:55:02,869-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 05:55:02,869-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 05:55:02,869-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 05:55:02,869-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 603c2b3e
>2018-05-29 05:55:02,869-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 05:55:02,882-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [596ceec3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:55:02,883-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [596ceec3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4e508659
>2018-05-29 05:55:02,886-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [596ceec3] FINISH, SetVdsStatusVDSCommand, log id: 4e508659
>2018-05-29 05:55:02,891-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [596ceec3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 596ceec3, Job ID: 70f26230-4c1c-4fc5-bc8e-b59a9af84850, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 05:55:02,896-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [596ceec3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d084fbe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 05:55:02,898-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [596ceec3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d084fbe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 05:55:02,909-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [5c4014be] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 05:55:02,910-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [5c4014be] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [54215fb9] Autorecovering 2 hosts
>2018-05-29 06:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [54215fb9] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:00:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5fdfff15] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5fdfff15] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5fdfff15] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5fdfff15] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:00:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5fdfff15] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4024b344
>2018-05-29 06:00:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5fdfff15] FINISH, SetVdsStatusVDSCommand, log id: 4024b344
>2018-05-29 06:00:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5fdfff15] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:00:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5fdfff15] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:00:00,083-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5fdfff15] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:00:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [48c5774f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:00:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [48c5774f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:00:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [48c5774f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:00:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [48c5774f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:00:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [48c5774f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 65c7d13c
>2018-05-29 06:00:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [48c5774f] FINISH, SetVdsStatusVDSCommand, log id: 65c7d13c
>2018-05-29 06:00:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [48c5774f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1f7171dd
>2018-05-29 06:00:00,104-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [48c5774f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1f7171dd
>2018-05-29 06:00:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [48c5774f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:00:00,106-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [48c5774f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:00:00,486-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [276a4e16] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7f3756ed
>2018-05-29 06:00:00,488-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [276a4e16] FINISH, GetHardwareInfoVDSCommand, log id: 7f3756ed
>2018-05-29 06:00:00,501-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [2099bee7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:00:00,510-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2d3d3edb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:00:00,515-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [1b0b0897] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:00:00,526-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [1b0b0897] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5bb00b57
>2018-05-29 06:00:00,903-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1b0b0897] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:00:00,903-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [1b0b0897] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5bb00b57
>2018-05-29 06:00:00,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1b0b0897] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:00:00,908-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [1b0b0897] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:00:00,911-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1b0b0897] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:00:00,912-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [1b0b0897] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:00:00,916-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1b0b0897] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:00:00,917-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [1b0b0897] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:00:00,920-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [1b0b0897] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:00:00,923-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1b0b0897] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1b0b0897, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:00:00,929-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2838ea2f
>2018-05-29 06:00:00,929-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:00:01,028-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:00:01,028-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:00:01,028-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:00:01,028-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:00:01,028-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2838ea2f
>2018-05-29 06:00:01,028-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:00:01,042-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [22548bb4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:00:01,043-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [22548bb4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 65afb6a5
>2018-05-29 06:00:01,045-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [22548bb4] FINISH, SetVdsStatusVDSCommand, log id: 65afb6a5
>2018-05-29 06:00:01,079-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [22548bb4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22548bb4, Job ID: 49a0ff3c-5478-4aef-9ba4-82a00630ee9a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:00:01,084-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [22548bb4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2d3d3edb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:00:01,086-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [22548bb4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2d3d3edb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:00:01,096-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [790e83c0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:00:01,097-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [790e83c0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:00:02,470-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [11ea5ef3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: c7298de
>2018-05-29 06:00:02,472-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [11ea5ef3] FINISH, GetHardwareInfoVDSCommand, log id: c7298de
>2018-05-29 06:00:02,486-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [17aea6b1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:00:02,490-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5dab5c7e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:00:02,496-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [43fa9c29] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:00:02,507-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [43fa9c29] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f919138
>2018-05-29 06:00:02,888-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43fa9c29] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:00:02,888-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [43fa9c29] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6f919138
>2018-05-29 06:00:02,892-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43fa9c29] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:00:02,892-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [43fa9c29] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:00:02,896-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43fa9c29] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:00:02,897-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [43fa9c29] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:00:02,901-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43fa9c29] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:00:02,902-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [43fa9c29] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:00:02,904-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [43fa9c29] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:00:02,907-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43fa9c29] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 43fa9c29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:00:02,913-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 69040d28
>2018-05-29 06:00:02,913-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:00:03,014-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-10) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:00:03,014-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:00:03,014-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:00:03,014-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:00:03,014-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] FINISH, ConnectStoragePoolVDSCommand, log id: 69040d28
>2018-05-29 06:00:03,014-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-10) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:00:03,028-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [18dd31c2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:00:03,029-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [18dd31c2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 497be59f
>2018-05-29 06:00:03,031-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [18dd31c2] FINISH, SetVdsStatusVDSCommand, log id: 497be59f
>2018-05-29 06:00:03,064-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18dd31c2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 18dd31c2, Job ID: cc260127-3980-4fa6-b7b3-48e2fe7e53f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:00:03,069-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18dd31c2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5dab5c7e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:00:03,071-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18dd31c2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5dab5c7e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:00:03,082-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [673544b4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:00:03,083-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [673544b4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:05:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4802a040] Autorecovering 2 hosts
>2018-05-29 06:05:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4802a040] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:05:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [556c3c99] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:05:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [556c3c99] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:05:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [556c3c99] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:05:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [556c3c99] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:05:00,054-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [556c3c99] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 470041ac
>2018-05-29 06:05:00,057-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [556c3c99] FINISH, SetVdsStatusVDSCommand, log id: 470041ac
>2018-05-29 06:05:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [556c3c99] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:05:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [556c3c99] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:05:00,060-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [556c3c99] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:05:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [486ec98e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:05:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [486ec98e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:05:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [486ec98e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:05:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [486ec98e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:05:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [486ec98e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7035db28
>2018-05-29 06:05:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [486ec98e] FINISH, SetVdsStatusVDSCommand, log id: 7035db28
>2018-05-29 06:05:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [486ec98e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1dfe717c
>2018-05-29 06:05:00,081-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [486ec98e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1dfe717c
>2018-05-29 06:05:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [486ec98e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:05:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [486ec98e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:05:00,667-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [85bfd40] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 27f0d21c
>2018-05-29 06:05:00,669-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [85bfd40] FINISH, GetHardwareInfoVDSCommand, log id: 27f0d21c
>2018-05-29 06:05:00,685-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [343016d0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:05:00,694-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6ee40a41] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:05:00,700-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [39abb8e1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:05:00,707-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [39abb8e1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6ac4973c
>2018-05-29 06:05:01,095-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [39abb8e1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:05:01,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [39abb8e1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6ac4973c
>2018-05-29 06:05:01,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [39abb8e1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:05:01,100-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [39abb8e1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:05:01,104-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [39abb8e1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:05:01,105-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [39abb8e1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:05:01,109-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [39abb8e1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:05:01,110-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [39abb8e1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:05:01,112-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [39abb8e1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:05:01,115-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [39abb8e1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 39abb8e1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:05:01,118-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 662c3b49
>2018-05-29 06:05:01,118-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:05:01,217-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:05:01,217-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:05:01,217-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:05:01,217-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:05:01,217-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 662c3b49
>2018-05-29 06:05:01,217-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:05:01,231-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [33e69171] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:05:01,232-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [33e69171] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f19d695
>2018-05-29 06:05:01,234-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [33e69171] FINISH, SetVdsStatusVDSCommand, log id: 3f19d695
>2018-05-29 06:05:01,266-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [33e69171] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 33e69171, Job ID: bf99815e-41c2-4582-8aa8-95015606dce9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:05:01,271-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [33e69171] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6ee40a41, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:05:01,273-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [33e69171] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6ee40a41, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:05:01,283-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [3245e4d2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:05:01,285-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [3245e4d2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:05:01,770-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [11158cee] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 73a341b4
>2018-05-29 06:05:01,772-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [11158cee] FINISH, GetHardwareInfoVDSCommand, log id: 73a341b4
>2018-05-29 06:05:01,785-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [635363e6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:05:01,789-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [604bca53] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:05:01,795-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [496a6752] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:05:01,803-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [496a6752] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: b4d1a04
>2018-05-29 06:05:02,172-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [496a6752] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:05:02,172-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [496a6752] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: b4d1a04
>2018-05-29 06:05:02,177-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [496a6752] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:05:02,177-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [496a6752] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:05:02,181-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [496a6752] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:05:02,182-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [496a6752] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:05:02,186-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [496a6752] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:05:02,187-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [496a6752] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:05:02,190-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [496a6752] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:05:02,193-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [496a6752] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 496a6752, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:05:02,200-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 44339707
>2018-05-29 06:05:02,200-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:05:02,298-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:05:02,298-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:05:02,298-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:05:02,298-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:05:02,298-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 44339707
>2018-05-29 06:05:02,298-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:05:02,313-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [69caf525] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:05:02,314-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [69caf525] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a08b43c
>2018-05-29 06:05:02,316-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [69caf525] FINISH, SetVdsStatusVDSCommand, log id: 5a08b43c
>2018-05-29 06:05:02,350-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [69caf525] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 69caf525, Job ID: 703a980c-74f2-4da5-b63b-c201eb48e6f8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:05:02,355-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [69caf525] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 604bca53, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:05:02,361-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [69caf525] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 604bca53, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:05:02,374-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [42bdccd2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:05:02,375-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [42bdccd2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1adc2354] Autorecovering 2 hosts
>2018-05-29 06:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1adc2354] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:10:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [22e72255] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:10:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [22e72255] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:10:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [22e72255] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:10:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [22e72255] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:10:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [22e72255] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 793252ad
>2018-05-29 06:10:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [22e72255] FINISH, SetVdsStatusVDSCommand, log id: 793252ad
>2018-05-29 06:10:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [22e72255] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:10:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [22e72255] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:10:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [22e72255] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:10:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [14189bca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:10:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [14189bca] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:10:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [14189bca] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:10:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [14189bca] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:10:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [14189bca] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a4fa78d
>2018-05-29 06:10:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [14189bca] FINISH, SetVdsStatusVDSCommand, log id: 5a4fa78d
>2018-05-29 06:10:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [14189bca] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 16d58975
>2018-05-29 06:10:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [14189bca] FINISH, SetHaMaintenanceModeVDSCommand, log id: 16d58975
>2018-05-29 06:10:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [14189bca] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:10:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [14189bca] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:10:01,898-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [423aba30] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 44eb1136
>2018-05-29 06:10:01,900-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [423aba30] FINISH, GetHardwareInfoVDSCommand, log id: 44eb1136
>2018-05-29 06:10:01,913-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [52f8dee6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:10:01,920-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [313209b7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:10:01,926-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [54d972b0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:10:01,937-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [54d972b0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4ad987f3
>2018-05-29 06:10:02,315-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [54d972b0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:10:02,315-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [54d972b0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4ad987f3
>2018-05-29 06:10:02,319-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [54d972b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:10:02,320-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [54d972b0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:10:02,324-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [54d972b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:10:02,324-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [54d972b0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:10:02,328-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [54d972b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:10:02,329-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [54d972b0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:10:02,332-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [54d972b0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:10:02,335-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [54d972b0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 54d972b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:10:02,338-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7103cc46
>2018-05-29 06:10:02,338-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:10:02,438-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:10:02,438-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:10:02,438-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:10:02,438-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:10:02,438-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7103cc46
>2018-05-29 06:10:02,438-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:10:02,452-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [221403fb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:10:02,453-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [221403fb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7eb99c08
>2018-05-29 06:10:02,455-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [221403fb] FINISH, SetVdsStatusVDSCommand, log id: 7eb99c08
>2018-05-29 06:10:02,486-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [221403fb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 221403fb, Job ID: 0bf7d187-5f70-458a-b701-3c8e9d5dd577, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:10:02,491-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [221403fb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 313209b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:10:02,493-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [221403fb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 313209b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:10:02,504-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7d88a9ae] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:10:02,505-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7d88a9ae] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:10:03,017-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [e2d209a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6bd5629d
>2018-05-29 06:10:03,019-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [e2d209a] FINISH, GetHardwareInfoVDSCommand, log id: 6bd5629d
>2018-05-29 06:10:03,032-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [71faf09a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:10:03,040-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [1c71bf2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:10:03,045-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [5362a0eb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:10:03,051-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [5362a0eb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 54057e63
>2018-05-29 06:10:03,420-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5362a0eb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:10:03,420-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [5362a0eb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 54057e63
>2018-05-29 06:10:03,424-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5362a0eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:10:03,424-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5362a0eb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:10:03,428-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5362a0eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:10:03,429-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5362a0eb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:10:03,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5362a0eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:10:03,433-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5362a0eb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:10:03,436-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [5362a0eb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:10:03,439-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5362a0eb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5362a0eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:10:03,441-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6eac8288
>2018-05-29 06:10:03,441-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:10:03,540-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:10:03,540-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:10:03,540-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:10:03,540-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:10:03,540-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6eac8288
>2018-05-29 06:10:03,540-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:10:03,553-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [f4f3366] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:10:03,554-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [f4f3366] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1c7d2494
>2018-05-29 06:10:03,556-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [f4f3366] FINISH, SetVdsStatusVDSCommand, log id: 1c7d2494
>2018-05-29 06:10:03,562-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [f4f3366] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: f4f3366, Job ID: 666d6af8-1bb8-46fd-9396-2bb85415c28c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:10:03,566-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [f4f3366] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1c71bf2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:10:03,568-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [f4f3366] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1c71bf2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:10:03,579-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [57c6b1a6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:10:03,580-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [57c6b1a6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:15:00,014-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3ea74022] Autorecovering 2 hosts
>2018-05-29 06:15:00,015-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3ea74022] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5bb94b13] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:15:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5bb94b13] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:15:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5bb94b13] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:15:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5bb94b13] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:15:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5bb94b13] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 11eacc22
>2018-05-29 06:15:00,083-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5bb94b13] FINISH, SetVdsStatusVDSCommand, log id: 11eacc22
>2018-05-29 06:15:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5bb94b13] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:15:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5bb94b13] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:15:00,086-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5bb94b13] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:15:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [a68e8c5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:15:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [a68e8c5] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:15:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [a68e8c5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:15:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [a68e8c5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:15:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [a68e8c5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f0122f4
>2018-05-29 06:15:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [a68e8c5] FINISH, SetVdsStatusVDSCommand, log id: f0122f4
>2018-05-29 06:15:00,105-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [a68e8c5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2b574e7d
>2018-05-29 06:15:00,107-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [a68e8c5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2b574e7d
>2018-05-29 06:15:00,107-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [a68e8c5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:15:00,108-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [a68e8c5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:15:01,247-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6d7213d5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 16a6d41b
>2018-05-29 06:15:01,249-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6d7213d5] FINISH, GetHardwareInfoVDSCommand, log id: 16a6d41b
>2018-05-29 06:15:01,263-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [aa074bf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:15:01,271-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [55369981] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:15:01,277-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [386708fe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:15:01,289-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [386708fe] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: c225796
>2018-05-29 06:15:02,665-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [386708fe] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:15:02,665-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [386708fe] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: c225796
>2018-05-29 06:15:02,669-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [386708fe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:15:02,670-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [386708fe] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:15:02,674-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [386708fe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:15:02,674-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [386708fe] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:15:02,678-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [386708fe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:15:02,679-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [386708fe] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:15:02,682-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [386708fe] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:15:02,685-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [386708fe] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 386708fe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:15:02,691-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 24ec33aa
>2018-05-29 06:15:02,691-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:15:02,787-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:15:02,787-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:15:02,787-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:15:02,788-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:15:02,788-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 24ec33aa
>2018-05-29 06:15:02,788-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:15:02,800-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [188715df] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:15:02,802-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [188715df] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 205aec82
>2018-05-29 06:15:02,804-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [188715df] FINISH, SetVdsStatusVDSCommand, log id: 205aec82
>2018-05-29 06:15:02,835-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [188715df] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 188715df, Job ID: 9321f4d1-5152-44d8-a06f-fea5bd9dca56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:15:02,840-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [188715df] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 55369981, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:15:02,842-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [188715df] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 55369981, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:15:02,853-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [aca1384] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:15:02,854-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [aca1384] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:15:03,129-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [4e18a92f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6a5ae3d6
>2018-05-29 06:15:03,130-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [4e18a92f] FINISH, GetHardwareInfoVDSCommand, log id: 6a5ae3d6
>2018-05-29 06:15:03,143-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [461c26a2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:15:03,151-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [4afcfe76] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:15:03,156-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [200faa58] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:15:03,167-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [200faa58] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: f5f6121
>2018-05-29 06:15:03,544-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [200faa58] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:15:03,544-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [200faa58] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: f5f6121
>2018-05-29 06:15:03,548-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [200faa58] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:15:03,549-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [200faa58] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:15:03,552-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [200faa58] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:15:03,553-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [200faa58] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:15:03,557-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [200faa58] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:15:03,558-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [200faa58] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:15:03,560-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [200faa58] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:15:03,563-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [200faa58] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 200faa58, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:15:03,566-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3f420dc6
>2018-05-29 06:15:03,566-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:15:03,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:15:03,664-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:15:03,664-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:15:03,664-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:15:03,664-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3f420dc6
>2018-05-29 06:15:03,664-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:15:03,677-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [6925b705] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:15:03,678-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6925b705] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 77c3ffc8
>2018-05-29 06:15:03,680-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6925b705] FINISH, SetVdsStatusVDSCommand, log id: 77c3ffc8
>2018-05-29 06:15:03,685-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6925b705] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6925b705, Job ID: c7303b68-5a57-4460-a68a-f656411a166c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:15:03,690-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6925b705] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4afcfe76, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:15:03,693-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6925b705] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4afcfe76, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:15:03,703-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [73ba7d1e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:15:03,705-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [73ba7d1e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:20:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [5e95e9de] Autorecovering 2 hosts
>2018-05-29 06:20:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [5e95e9de] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:20:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [66f0e607] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:20:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [66f0e607] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:20:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [66f0e607] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:20:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [66f0e607] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:20:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [66f0e607] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 482b1666
>2018-05-29 06:20:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [66f0e607] FINISH, SetVdsStatusVDSCommand, log id: 482b1666
>2018-05-29 06:20:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [66f0e607] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:20:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [66f0e607] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:20:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [66f0e607] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:20:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [35ca54e6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:20:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [35ca54e6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:20:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [35ca54e6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:20:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [35ca54e6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:20:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [35ca54e6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4d684cb3
>2018-05-29 06:20:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [35ca54e6] FINISH, SetVdsStatusVDSCommand, log id: 4d684cb3
>2018-05-29 06:20:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [35ca54e6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3630c119
>2018-05-29 06:20:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [35ca54e6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3630c119
>2018-05-29 06:20:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [35ca54e6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:20:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [35ca54e6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:20:00,554-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2e5603a6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4893ab17
>2018-05-29 06:20:00,556-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2e5603a6] FINISH, GetHardwareInfoVDSCommand, log id: 4893ab17
>2018-05-29 06:20:00,569-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [3f85bbb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:20:00,577-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [4350d401] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:20:00,582-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [39687d9d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:20:00,594-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [39687d9d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22541ee2
>2018-05-29 06:20:00,967-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39687d9d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:20:00,967-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [39687d9d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 22541ee2
>2018-05-29 06:20:00,971-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39687d9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:20:00,972-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [39687d9d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:20:00,976-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39687d9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:20:00,977-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [39687d9d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:20:00,981-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39687d9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:20:00,982-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [39687d9d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:20:00,984-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [39687d9d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:20:00,987-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39687d9d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 39687d9d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:20:00,993-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 9c13824
>2018-05-29 06:20:00,993-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:20:01,091-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:20:01,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:20:01,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:20:01,091-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:20:01,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 9c13824
>2018-05-29 06:20:01,091-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:20:01,105-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [ecf004c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:20:01,107-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [ecf004c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 832652b
>2018-05-29 06:20:01,109-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [ecf004c] FINISH, SetVdsStatusVDSCommand, log id: 832652b
>2018-05-29 06:20:01,141-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ecf004c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: ecf004c, Job ID: 092cce54-4018-433e-8d82-0de9a65ab100, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:20:01,146-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ecf004c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4350d401, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:20:01,149-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ecf004c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4350d401, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:20:01,160-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [313925f9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:20:01,161-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [313925f9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:20:01,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5475308b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 22eca576
>2018-05-29 06:20:01,286-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5475308b] FINISH, GetHardwareInfoVDSCommand, log id: 22eca576
>2018-05-29 06:20:01,300-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [35b8d1f6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:20:01,309-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [175cf15] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:20:01,314-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [30af2c43] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:20:01,321-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [30af2c43] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4f987cfb
>2018-05-29 06:20:01,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30af2c43] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:20:01,704-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [30af2c43] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4f987cfb
>2018-05-29 06:20:01,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30af2c43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:20:01,709-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [30af2c43] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:20:01,713-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30af2c43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:20:01,713-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [30af2c43] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:20:01,717-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30af2c43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:20:01,718-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [30af2c43] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:20:01,721-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [30af2c43] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:20:01,724-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30af2c43] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 30af2c43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:20:01,730-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7a5ba819
>2018-05-29 06:20:01,730-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:20:01,828-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:20:01,828-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:20:01,828-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:20:01,828-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:20:01,828-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7a5ba819
>2018-05-29 06:20:01,828-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:20:01,842-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [384f63f0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:20:01,843-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [384f63f0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 75eb233a
>2018-05-29 06:20:01,846-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [384f63f0] FINISH, SetVdsStatusVDSCommand, log id: 75eb233a
>2018-05-29 06:20:01,877-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [384f63f0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 384f63f0, Job ID: 31efce1a-0bc4-4af7-8bf5-efb63d80c390, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:20:01,882-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [384f63f0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 175cf15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:20:01,884-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [384f63f0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 175cf15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:20:01,895-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [31cc5978] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:20:01,896-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [31cc5978] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:25:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [534c1ee6] Autorecovering 2 hosts
>2018-05-29 06:25:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [534c1ee6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:25:00,033-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3ac45e5d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:25:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3ac45e5d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:25:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3ac45e5d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:25:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3ac45e5d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:25:00,041-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3ac45e5d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3793a355
>2018-05-29 06:25:00,043-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3ac45e5d] FINISH, SetVdsStatusVDSCommand, log id: 3793a355
>2018-05-29 06:25:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3ac45e5d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:25:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3ac45e5d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:25:00,046-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3ac45e5d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:25:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [43c5efb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:25:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [43c5efb] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:25:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [43c5efb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:25:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [43c5efb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:25:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [43c5efb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 20ebc2d1
>2018-05-29 06:25:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [43c5efb] FINISH, SetVdsStatusVDSCommand, log id: 20ebc2d1
>2018-05-29 06:25:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [43c5efb] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3a347812
>2018-05-29 06:25:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [43c5efb] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3a347812
>2018-05-29 06:25:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [43c5efb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:25:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [43c5efb] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:25:01,825-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [10669cef] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5965d059
>2018-05-29 06:25:01,826-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [10669cef] FINISH, GetHardwareInfoVDSCommand, log id: 5965d059
>2018-05-29 06:25:01,840-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [18ca4696] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:25:01,844-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [18e4b306] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:25:01,849-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [59099b72] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:25:01,861-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [59099b72] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 61243277
>2018-05-29 06:25:02,583-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [325068d3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 455dea02
>2018-05-29 06:25:02,585-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [325068d3] FINISH, GetHardwareInfoVDSCommand, log id: 455dea02
>2018-05-29 06:25:02,598-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [1d37f250] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:25:02,606-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [65defd17] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:25:02,612-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [6c755c5c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:25:02,667-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [6c755c5c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2fe0916
>2018-05-29 06:25:03,048-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c755c5c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:25:03,048-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [6c755c5c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2fe0916
>2018-05-29 06:25:03,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c755c5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:25:03,052-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6c755c5c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:25:03,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c755c5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:25:03,057-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6c755c5c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:25:03,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c755c5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:25:03,061-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6c755c5c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:25:03,064-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [6c755c5c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:25:03,067-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c755c5c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6c755c5c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:25:03,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 210c9523
>2018-05-29 06:25:03,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:25:03,171-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:25:03,171-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:25:03,171-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:25:03,171-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:25:03,171-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 210c9523
>2018-05-29 06:25:03,171-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:25:03,184-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [1daa79ff] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:25:03,186-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1daa79ff] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 409fea01
>2018-05-29 06:25:03,188-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1daa79ff] FINISH, SetVdsStatusVDSCommand, log id: 409fea01
>2018-05-29 06:25:03,221-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1daa79ff] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1daa79ff, Job ID: 68882bb5-3369-47f6-b6cc-3269be0df60c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:25:03,226-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1daa79ff] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 65defd17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:25:03,228-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1daa79ff] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 65defd17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:25:03,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [59099b72] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:25:03,230-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [59099b72] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 61243277
>2018-05-29 06:25:03,234-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [59099b72] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:25:03,235-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [59099b72] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:25:03,239-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [59099b72] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:25:03,240-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [47f0ae07] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:25:03,240-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [59099b72] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:25:03,241-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [47f0ae07] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:25:03,244-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [59099b72] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:25:03,245-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [59099b72] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:25:03,248-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [59099b72] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:25:03,251-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [59099b72] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 59099b72, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:25:03,257-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2c326be5
>2018-05-29 06:25:03,257-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:25:03,354-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:25:03,354-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:25:03,354-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:25:03,354-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:25:03,354-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2c326be5
>2018-05-29 06:25:03,354-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:25:03,368-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [1a11e440] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:25:03,369-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1a11e440] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 108de41
>2018-05-29 06:25:03,372-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1a11e440] FINISH, SetVdsStatusVDSCommand, log id: 108de41
>2018-05-29 06:25:03,380-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1a11e440] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1a11e440, Job ID: cfee0ab8-cf8f-4746-a8d3-35cf66068b6b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:25:03,385-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1a11e440] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 18e4b306, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:25:03,387-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1a11e440] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 18e4b306, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:25:03,397-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [7d7bfd00] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:25:03,398-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [7d7bfd00] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:30:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [663e60b6] Autorecovering 2 hosts
>2018-05-29 06:30:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [663e60b6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:30:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d890adc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:30:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d890adc] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:30:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d890adc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:30:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d890adc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:30:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1d890adc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17640c7
>2018-05-29 06:30:00,081-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1d890adc] FINISH, SetVdsStatusVDSCommand, log id: 17640c7
>2018-05-29 06:30:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d890adc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:30:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d890adc] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:30:00,084-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1d890adc] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:30:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7e739db3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:30:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7e739db3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:30:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7e739db3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:30:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7e739db3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:30:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7e739db3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6fc78c9
>2018-05-29 06:30:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7e739db3] FINISH, SetVdsStatusVDSCommand, log id: 6fc78c9
>2018-05-29 06:30:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [7e739db3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 48766094
>2018-05-29 06:30:00,103-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [7e739db3] FINISH, SetHaMaintenanceModeVDSCommand, log id: 48766094
>2018-05-29 06:30:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7e739db3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:30:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7e739db3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:30:00,939-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [75538bbf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: ba7758f
>2018-05-29 06:30:00,941-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [75538bbf] FINISH, GetHardwareInfoVDSCommand, log id: ba7758f
>2018-05-29 06:30:00,953-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [6abc3387] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:30:00,962-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [682dc8a0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:30:00,967-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2a814799] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:30:01,025-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2a814799] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2f8e38d0
>2018-05-29 06:30:01,031-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [692e6853] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2cb741e4
>2018-05-29 06:30:01,033-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [692e6853] FINISH, GetHardwareInfoVDSCommand, log id: 2cb741e4
>2018-05-29 06:30:01,047-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [27bc83c2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:30:01,055-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [6544d668] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:30:01,060-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [647096d6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:30:01,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [647096d6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1da61315
>2018-05-29 06:30:01,407-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2a814799] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:30:01,407-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2a814799] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2f8e38d0
>2018-05-29 06:30:01,411-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2a814799] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:30:01,412-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2a814799] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:30:01,415-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2a814799] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:30:01,416-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2a814799] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:30:01,420-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2a814799] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:30:01,420-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2a814799] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:30:01,423-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2a814799] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:30:01,426-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2a814799] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2a814799, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:30:01,432-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7d75b5d2
>2018-05-29 06:30:01,432-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:30:01,440-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [647096d6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:30:01,440-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [647096d6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1da61315
>2018-05-29 06:30:01,443-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [647096d6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:30:01,444-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [647096d6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:30:01,448-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [647096d6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:30:01,449-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [647096d6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:30:01,452-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [647096d6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:30:01,453-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [647096d6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:30:01,456-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [647096d6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:30:01,458-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [647096d6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 647096d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:30:01,533-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:30:01,533-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:30:01,533-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:30:01,533-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:30:01,533-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7d75b5d2
>2018-05-29 06:30:01,533-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:30:01,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 11b564c8
>2018-05-29 06:30:01,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:30:01,547-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [141c773d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:30:01,548-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [141c773d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46d5b247
>2018-05-29 06:30:01,551-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [141c773d] FINISH, SetVdsStatusVDSCommand, log id: 46d5b247
>2018-05-29 06:30:01,583-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [141c773d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 141c773d, Job ID: cd5ba4ee-2c26-43f2-b45a-80460d40e574, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:30:01,587-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [141c773d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 682dc8a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:30:01,590-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [141c773d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 682dc8a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:30:01,600-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [6dcf6495] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:30:01,601-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [6dcf6495] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:30:01,636-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-10) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:30:01,636-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:30:01,636-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:30:01,636-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:30:01,636-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] FINISH, ConnectStoragePoolVDSCommand, log id: 11b564c8
>2018-05-29 06:30:01,636-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-10) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:30:01,648-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [475357dc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:30:01,650-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [475357dc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 38b07259
>2018-05-29 06:30:01,651-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [475357dc] FINISH, SetVdsStatusVDSCommand, log id: 38b07259
>2018-05-29 06:30:01,660-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [475357dc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 475357dc, Job ID: d49134a0-071d-4caf-bea3-9455a168b756, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:30:01,664-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [475357dc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6544d668, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:30:01,666-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [475357dc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6544d668, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:30:01,676-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [4d1ba88d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:30:01,678-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4d1ba88d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:35:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [1ed9314d] Autorecovering 2 hosts
>2018-05-29 06:35:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [1ed9314d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:35:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4789fd9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:35:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4789fd9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:35:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4789fd9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:35:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4789fd9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:35:00,045-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4789fd9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f6085a4
>2018-05-29 06:35:00,047-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4789fd9] FINISH, SetVdsStatusVDSCommand, log id: 5f6085a4
>2018-05-29 06:35:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4789fd9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:35:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4789fd9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:35:00,049-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [4789fd9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:35:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [601d822b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:35:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [601d822b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:35:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [601d822b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:35:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [601d822b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:35:00,062-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [601d822b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 58cbe2c0
>2018-05-29 06:35:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [601d822b] FINISH, SetVdsStatusVDSCommand, log id: 58cbe2c0
>2018-05-29 06:35:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [601d822b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2f256356
>2018-05-29 06:35:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [601d822b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2f256356
>2018-05-29 06:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [601d822b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:35:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [601d822b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:35:02,368-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7cd4ff8d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 22fc88f4
>2018-05-29 06:35:02,370-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7cd4ff8d] FINISH, GetHardwareInfoVDSCommand, log id: 22fc88f4
>2018-05-29 06:35:02,384-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [6cb6b724] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:35:02,388-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [399f2f32] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:35:02,393-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [c39f210] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:35:02,400-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [c39f210] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 267e9ecd
>2018-05-29 06:35:02,570-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [25c8bfe0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2439facb
>2018-05-29 06:35:02,572-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [25c8bfe0] FINISH, GetHardwareInfoVDSCommand, log id: 2439facb
>2018-05-29 06:35:02,585-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [1c7752c0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:35:02,594-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [2d66d31d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:35:02,599-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [78de10ae] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:35:02,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [78de10ae] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1fe9cebd
>2018-05-29 06:35:02,782-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [c39f210] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:35:02,782-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [c39f210] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 267e9ecd
>2018-05-29 06:35:02,786-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [c39f210] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:35:02,786-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [c39f210] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:35:02,790-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [c39f210] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:35:02,791-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [c39f210] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:35:02,794-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [c39f210] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:35:02,795-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [c39f210] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:35:02,798-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [c39f210] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:35:02,800-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [c39f210] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: c39f210, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:35:02,807-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 37d00051
>2018-05-29 06:35:02,807-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:35:02,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:35:02,907-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:35:02,907-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:35:02,907-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:35:02,907-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 37d00051
>2018-05-29 06:35:02,907-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:35:02,920-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [25abda3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:35:02,921-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [25abda3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c7a79d4
>2018-05-29 06:35:02,923-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [25abda3] FINISH, SetVdsStatusVDSCommand, log id: 7c7a79d4
>2018-05-29 06:35:02,956-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [25abda3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 25abda3, Job ID: c8c3ce74-f088-4795-9867-085d7ce240d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:35:02,960-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [25abda3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 399f2f32, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:35:02,962-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [25abda3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 399f2f32, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:35:02,972-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [1f23ab24] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:35:02,974-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1f23ab24] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:35:02,976-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78de10ae] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:35:02,976-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [78de10ae] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1fe9cebd
>2018-05-29 06:35:02,980-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78de10ae] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:35:02,981-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [78de10ae] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:35:02,984-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78de10ae] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:35:02,985-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [78de10ae] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:35:02,988-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78de10ae] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:35:02,989-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [78de10ae] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:35:02,993-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [78de10ae] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:35:02,995-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78de10ae] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 78de10ae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:35:03,002-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7966d5c9
>2018-05-29 06:35:03,002-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:35:03,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:35:03,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:35:03,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:35:03,099-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:35:03,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7966d5c9
>2018-05-29 06:35:03,099-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:35:03,111-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [378f75de] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:35:03,113-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [378f75de] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4cff1ee8
>2018-05-29 06:35:03,115-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [378f75de] FINISH, SetVdsStatusVDSCommand, log id: 4cff1ee8
>2018-05-29 06:35:03,143-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [378f75de] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 378f75de, Job ID: 6f531223-897f-49e3-958a-b7cbeef26182, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:35:03,148-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [378f75de] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2d66d31d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:35:03,151-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [378f75de] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2d66d31d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:35:03,161-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [ee4525a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:35:03,162-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [ee4525a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [6dba9def] Autorecovering 2 hosts
>2018-05-29 06:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [6dba9def] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:40:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5bd50d79] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5bd50d79] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5bd50d79] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5bd50d79] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:40:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5bd50d79] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 36fea794
>2018-05-29 06:40:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5bd50d79] FINISH, SetVdsStatusVDSCommand, log id: 36fea794
>2018-05-29 06:40:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5bd50d79] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:40:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5bd50d79] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:40:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [5bd50d79] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:40:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [695167f9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:40:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [695167f9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:40:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [695167f9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:40:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [695167f9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:40:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [695167f9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f47aa5f
>2018-05-29 06:40:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [695167f9] FINISH, SetVdsStatusVDSCommand, log id: 3f47aa5f
>2018-05-29 06:40:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [695167f9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5727c42b
>2018-05-29 06:40:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [695167f9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5727c42b
>2018-05-29 06:40:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [695167f9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:40:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [695167f9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:40:00,590-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1c85aa4f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 62c44616
>2018-05-29 06:40:00,592-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1c85aa4f] FINISH, GetHardwareInfoVDSCommand, log id: 62c44616
>2018-05-29 06:40:00,605-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [5ee27b1d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:40:00,613-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [e36636d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:40:00,618-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3a37b3c0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:40:00,629-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3a37b3c0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 67141f80
>2018-05-29 06:40:00,793-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [39c925f9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 788bade2
>2018-05-29 06:40:00,795-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [39c925f9] FINISH, GetHardwareInfoVDSCommand, log id: 788bade2
>2018-05-29 06:40:00,808-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [1ffa3ca8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:40:00,812-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [3d774928] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:40:00,817-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [55f6d92e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:40:00,823-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [55f6d92e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3891cdf6
>2018-05-29 06:40:01,012-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3a37b3c0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:40:01,012-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3a37b3c0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 67141f80
>2018-05-29 06:40:01,016-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3a37b3c0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:40:01,016-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3a37b3c0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:40:01,020-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3a37b3c0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:40:01,021-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3a37b3c0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:40:01,025-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3a37b3c0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:40:01,026-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3a37b3c0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:40:01,029-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3a37b3c0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:40:01,031-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3a37b3c0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3a37b3c0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:40:01,038-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2beb5338
>2018-05-29 06:40:01,039-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:40:01,135-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:40:01,135-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:40:01,135-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:40:01,135-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:40:01,135-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2beb5338
>2018-05-29 06:40:01,135-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:40:01,148-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [610611ca] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:40:01,149-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [610611ca] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2226ed5d
>2018-05-29 06:40:01,151-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [610611ca] FINISH, SetVdsStatusVDSCommand, log id: 2226ed5d
>2018-05-29 06:40:01,183-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [610611ca] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 610611ca, Job ID: 1a679d65-4457-489b-8ecb-bd39b8bb59e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:40:01,187-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [610611ca] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: e36636d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:40:01,189-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [610611ca] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: e36636d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:40:01,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55f6d92e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:40:01,198-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [55f6d92e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3891cdf6
>2018-05-29 06:40:01,201-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [10853814] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:40:01,202-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [10853814] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:40:01,203-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55f6d92e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:40:01,203-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [55f6d92e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:40:01,207-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55f6d92e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:40:01,208-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [55f6d92e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:40:01,211-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55f6d92e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:40:01,212-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [55f6d92e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:40:01,215-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [55f6d92e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:40:01,218-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55f6d92e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 55f6d92e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:40:01,225-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 250b76c5
>2018-05-29 06:40:01,225-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:40:01,324-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:40:01,324-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:40:01,324-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:40:01,324-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:40:01,324-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 250b76c5
>2018-05-29 06:40:01,325-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:40:01,337-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [14a76402] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:40:01,338-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [14a76402] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6b4f90db
>2018-05-29 06:40:01,341-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [14a76402] FINISH, SetVdsStatusVDSCommand, log id: 6b4f90db
>2018-05-29 06:40:01,349-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14a76402] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 14a76402, Job ID: 75212b5b-40b2-4e5d-9ef9-cae8b439b8d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:40:01,354-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14a76402] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3d774928, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:40:01,356-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14a76402] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3d774928, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:40:01,366-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [13b6194] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:40:01,367-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [13b6194] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [60370174] Autorecovering 2 hosts
>2018-05-29 06:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [60370174] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:45:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [625b28c9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [625b28c9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [625b28c9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [625b28c9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:45:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [625b28c9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7fb347bc
>2018-05-29 06:45:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [625b28c9] FINISH, SetVdsStatusVDSCommand, log id: 7fb347bc
>2018-05-29 06:45:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [625b28c9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:45:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [625b28c9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:45:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [625b28c9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:45:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1905e3fd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:45:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1905e3fd] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:45:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1905e3fd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:45:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1905e3fd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:45:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1905e3fd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 200ebc51
>2018-05-29 06:45:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1905e3fd] FINISH, SetVdsStatusVDSCommand, log id: 200ebc51
>2018-05-29 06:45:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [1905e3fd] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1a2eba86
>2018-05-29 06:45:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [1905e3fd] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1a2eba86
>2018-05-29 06:45:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1905e3fd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:45:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1905e3fd] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:45:01,869-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4f0b9e5d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: c9600f4
>2018-05-29 06:45:01,871-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4f0b9e5d] FINISH, GetHardwareInfoVDSCommand, log id: c9600f4
>2018-05-29 06:45:01,886-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [69ffd17c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:45:01,894-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [65d6a0c3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:45:01,899-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5eb06301] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:45:01,957-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5eb06301] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 540ba70f
>2018-05-29 06:45:02,243-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [37b5c1cd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 48d1e8ce
>2018-05-29 06:45:02,245-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [37b5c1cd] FINISH, GetHardwareInfoVDSCommand, log id: 48d1e8ce
>2018-05-29 06:45:02,259-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [272e8c7e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:45:02,264-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [400918e3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:45:02,269-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2d4dc7d8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:45:02,276-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2d4dc7d8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7fcba56e
>2018-05-29 06:45:02,338-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eb06301] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:45:02,338-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5eb06301] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 540ba70f
>2018-05-29 06:45:02,343-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eb06301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:45:02,343-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5eb06301] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:45:02,347-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eb06301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:45:02,348-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5eb06301] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:45:02,351-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eb06301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:45:02,352-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5eb06301] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:45:02,354-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5eb06301] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:45:02,357-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eb06301] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5eb06301, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:45:02,364-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7ca808d3
>2018-05-29 06:45:02,364-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:45:02,463-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:45:02,463-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:45:02,463-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:45:02,463-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:45:02,463-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7ca808d3
>2018-05-29 06:45:02,463-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:45:02,476-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [5d8989ed] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:45:02,478-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5d8989ed] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 290f833b
>2018-05-29 06:45:02,480-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5d8989ed] FINISH, SetVdsStatusVDSCommand, log id: 290f833b
>2018-05-29 06:45:02,514-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5d8989ed] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5d8989ed, Job ID: ebaf20b4-1b02-4d93-98d1-1ced8ad4b32f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:45:02,519-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5d8989ed] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 65d6a0c3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:45:02,521-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5d8989ed] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 65d6a0c3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:45:02,532-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [28186840] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:45:02,533-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [28186840] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:45:04,589-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d4dc7d8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:45:04,590-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2d4dc7d8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7fcba56e
>2018-05-29 06:45:04,594-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d4dc7d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:45:04,594-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2d4dc7d8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:45:04,598-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d4dc7d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:45:04,599-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2d4dc7d8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:45:04,603-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d4dc7d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:45:04,603-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2d4dc7d8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:45:04,607-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2d4dc7d8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:45:04,609-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d4dc7d8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2d4dc7d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:45:04,612-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4e3f66f6
>2018-05-29 06:45:04,612-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:45:04,709-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:45:04,709-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:45:04,709-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:45:04,709-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:45:04,709-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4e3f66f6
>2018-05-29 06:45:04,709-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:45:04,723-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [5fda4dc3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:45:04,724-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5fda4dc3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43866b1
>2018-05-29 06:45:04,726-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5fda4dc3] FINISH, SetVdsStatusVDSCommand, log id: 43866b1
>2018-05-29 06:45:04,756-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5fda4dc3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5fda4dc3, Job ID: 1548856c-039e-494b-8706-96c4cf43ce89, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:45:04,761-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5fda4dc3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 400918e3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:45:04,763-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5fda4dc3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 400918e3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:45:04,774-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [b8ee3aa] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:45:04,775-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [b8ee3aa] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [627e71c0] Autorecovering 2 hosts
>2018-05-29 06:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [627e71c0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:50:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50a822ab] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:50:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50a822ab] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:50:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50a822ab] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:50:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50a822ab] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:50:00,046-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [50a822ab] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7673280f
>2018-05-29 06:50:00,048-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [50a822ab] FINISH, SetVdsStatusVDSCommand, log id: 7673280f
>2018-05-29 06:50:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50a822ab] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:50:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50a822ab] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:50:00,051-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [50a822ab] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:50:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52183274] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:50:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52183274] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:50:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52183274] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:50:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52183274] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:50:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [52183274] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 666a08f6
>2018-05-29 06:50:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [52183274] FINISH, SetVdsStatusVDSCommand, log id: 666a08f6
>2018-05-29 06:50:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [52183274] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 79762ae4
>2018-05-29 06:50:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [52183274] FINISH, SetHaMaintenanceModeVDSCommand, log id: 79762ae4
>2018-05-29 06:50:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52183274] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:50:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52183274] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:50:02,455-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [46f097d4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4e2bffd3
>2018-05-29 06:50:02,457-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [46f097d4] FINISH, GetHardwareInfoVDSCommand, log id: 4e2bffd3
>2018-05-29 06:50:02,471-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [697fbceb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:50:02,475-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [6292c433] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:50:02,480-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5b12e033] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:50:02,492-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5b12e033] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2ebdfdb4
>2018-05-29 06:50:02,867-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5b12e033] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:50:02,867-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5b12e033] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2ebdfdb4
>2018-05-29 06:50:02,871-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5b12e033] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:50:02,872-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5b12e033] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:50:02,876-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5b12e033] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:50:02,877-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5b12e033] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:50:02,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5b12e033] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:50:02,881-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5b12e033] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:50:02,926-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5b12e033] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:50:02,930-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5b12e033] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5b12e033, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:50:02,937-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 48e7b12f
>2018-05-29 06:50:02,937-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:50:03,037-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:50:03,037-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:50:03,037-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:50:03,037-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:50:03,037-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 48e7b12f
>2018-05-29 06:50:03,037-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:50:03,052-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [7a2f618e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:50:03,054-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7a2f618e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7628fbaf
>2018-05-29 06:50:03,056-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7a2f618e] FINISH, SetVdsStatusVDSCommand, log id: 7628fbaf
>2018-05-29 06:50:03,087-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a2f618e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7a2f618e, Job ID: 556cc801-306d-4ac6-bb77-151ceb469ba7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:50:03,092-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a2f618e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6292c433, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:50:03,094-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a2f618e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6292c433, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:50:03,105-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2aed90aa] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:50:03,106-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2aed90aa] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:50:03,141-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5349faef] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5a220e7a
>2018-05-29 06:50:03,143-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5349faef] FINISH, GetHardwareInfoVDSCommand, log id: 5a220e7a
>2018-05-29 06:50:03,156-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [46b7bfaa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:50:03,165-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [3ef776c5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:50:03,170-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2c890233] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:50:03,183-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2c890233] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 45a8e617
>2018-05-29 06:50:03,563-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2c890233] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:50:03,563-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2c890233] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 45a8e617
>2018-05-29 06:50:03,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2c890233] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:50:03,568-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2c890233] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:50:03,572-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2c890233] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:50:03,573-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2c890233] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:50:03,577-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2c890233] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:50:03,577-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2c890233] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:50:03,581-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2c890233] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:50:03,583-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2c890233] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2c890233, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:50:03,586-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4c5fa7b
>2018-05-29 06:50:03,586-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:50:03,684-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:50:03,684-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:50:03,684-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:50:03,684-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:50:03,684-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4c5fa7b
>2018-05-29 06:50:03,684-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:50:03,697-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [269befa6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:50:03,698-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [269befa6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17cf0b62
>2018-05-29 06:50:03,701-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [269befa6] FINISH, SetVdsStatusVDSCommand, log id: 17cf0b62
>2018-05-29 06:50:03,706-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [269befa6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 269befa6, Job ID: 0cf880b8-84b1-45dd-ad32-ba6841d563ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:50:03,711-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [269befa6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3ef776c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:50:03,713-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [269befa6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3ef776c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:50:03,724-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [79958cd6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:50:03,725-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [79958cd6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:54:42,893-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler5) [4478f03c] Attempting to update VMs/Templates Ovf.
>2018-05-29 06:54:42,896-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 06:54:42,896-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 06:54:42,896-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:54:42,901-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 06:54:42,904-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 06:54:42,904-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 06:54:42,904-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 06:54:42,904-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 06:54:42,905-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 06:54:42,905-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 06:55:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [70406a73] Autorecovering 2 hosts
>2018-05-29 06:55:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [70406a73] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:55:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [9acbffd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:55:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [9acbffd] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:55:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [9acbffd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:55:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [9acbffd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:55:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [9acbffd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1354c90e
>2018-05-29 06:55:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [9acbffd] FINISH, SetVdsStatusVDSCommand, log id: 1354c90e
>2018-05-29 06:55:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [9acbffd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:55:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [9acbffd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 06:55:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [9acbffd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:55:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [214409f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:55:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [214409f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 06:55:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [214409f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:55:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [214409f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:55:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [214409f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6f291a09
>2018-05-29 06:55:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [214409f] FINISH, SetVdsStatusVDSCommand, log id: 6f291a09
>2018-05-29 06:55:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [214409f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 62d69250
>2018-05-29 06:55:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [214409f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 62d69250
>2018-05-29 06:55:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [214409f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 06:55:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [214409f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 06:55:00,760-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [67bce0e7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 72f619d
>2018-05-29 06:55:00,761-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [67bce0e7] FINISH, GetHardwareInfoVDSCommand, log id: 72f619d
>2018-05-29 06:55:00,776-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [797ad905] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:55:00,784-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [177b6c9f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:55:00,789-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3afec7c2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:55:00,800-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3afec7c2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 659d0c20
>2018-05-29 06:55:01,174-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3afec7c2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:55:01,174-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3afec7c2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 659d0c20
>2018-05-29 06:55:01,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3afec7c2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:55:01,179-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3afec7c2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:55:01,183-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3afec7c2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:55:01,184-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3afec7c2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:55:01,187-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3afec7c2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:55:01,188-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3afec7c2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:55:01,191-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3afec7c2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:55:01,193-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3afec7c2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3afec7c2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:55:01,199-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 76561eca
>2018-05-29 06:55:01,199-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:55:01,297-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:55:01,297-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:55:01,297-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:55:01,297-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:55:01,297-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 76561eca
>2018-05-29 06:55:01,297-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:55:01,310-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [494b55ee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:55:01,311-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [494b55ee] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6c75d74f
>2018-05-29 06:55:01,313-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [494b55ee] FINISH, SetVdsStatusVDSCommand, log id: 6c75d74f
>2018-05-29 06:55:01,345-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [494b55ee] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 494b55ee, Job ID: c8fd0afb-0c2b-41ba-b40f-e26617ba890d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:55:01,350-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [494b55ee] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 177b6c9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:55:01,352-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [494b55ee] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 177b6c9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:55:01,359-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [78a111a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 555cb372
>2018-05-29 06:55:01,361-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [78a111a] FINISH, GetHardwareInfoVDSCommand, log id: 555cb372
>2018-05-29 06:55:01,364-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [201d2106] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 06:55:01,365-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [201d2106] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 06:55:01,396-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [4c6827c3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:55:01,404-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [528e722c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:55:01,410-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2658a34] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 06:55:01,469-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2658a34] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 63cbb68a
>2018-05-29 06:55:01,847-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2658a34] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 06:55:01,847-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2658a34] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 63cbb68a
>2018-05-29 06:55:01,851-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2658a34] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:55:01,852-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2658a34] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:55:01,856-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2658a34] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:55:01,857-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2658a34] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:55:01,861-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2658a34] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 06:55:01,861-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2658a34] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 06:55:01,864-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2658a34] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 06:55:01,867-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2658a34] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2658a34, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 06:55:01,874-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 85b3b79
>2018-05-29 06:55:01,874-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 06:55:01,969-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:55:01,969-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 06:55:01,969-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 06:55:01,969-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 06:55:01,969-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 85b3b79
>2018-05-29 06:55:01,969-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 06:55:01,983-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [eee1357] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:55:01,985-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [eee1357] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 275d4bb6
>2018-05-29 06:55:01,987-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [eee1357] FINISH, SetVdsStatusVDSCommand, log id: 275d4bb6
>2018-05-29 06:55:02,019-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [eee1357] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: eee1357, Job ID: 2fa79f6d-d0f2-4aaf-b9d8-a9d5a64c2c48, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 06:55:02,023-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [eee1357] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 528e722c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 06:55:02,025-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [eee1357] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 528e722c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 06:55:02,037-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [25aaff6f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 06:55:02,038-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [25aaff6f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [45ebefb1] Autorecovering 2 hosts
>2018-05-29 07:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [45ebefb1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:00:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [55fb9889] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [55fb9889] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [55fb9889] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [55fb9889] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:00:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [55fb9889] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4958bfab
>2018-05-29 07:00:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [55fb9889] FINISH, SetVdsStatusVDSCommand, log id: 4958bfab
>2018-05-29 07:00:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [55fb9889] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:00:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [55fb9889] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:00:00,083-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [55fb9889] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:00:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6cd2f160] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6cd2f160] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6cd2f160] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6cd2f160] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:00:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6cd2f160] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33841fb0
>2018-05-29 07:00:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6cd2f160] FINISH, SetVdsStatusVDSCommand, log id: 33841fb0
>2018-05-29 07:00:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [6cd2f160] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 413fc040
>2018-05-29 07:00:00,104-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [6cd2f160] FINISH, SetHaMaintenanceModeVDSCommand, log id: 413fc040
>2018-05-29 07:00:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6cd2f160] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:00:00,105-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6cd2f160] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:00:02,249-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [338a57ab] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 44b0ec8c
>2018-05-29 07:00:02,251-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [338a57ab] FINISH, GetHardwareInfoVDSCommand, log id: 44b0ec8c
>2018-05-29 07:00:02,264-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [66e475ea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:00:02,273-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [2cd74724] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:00:02,278-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [19cab49e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:00:02,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [19cab49e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 12b6e7a4
>2018-05-29 07:00:02,651-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19cab49e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:00:02,651-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [19cab49e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 12b6e7a4
>2018-05-29 07:00:02,655-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19cab49e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:00:02,656-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [19cab49e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:00:02,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19cab49e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:00:02,660-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [19cab49e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:00:02,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19cab49e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:00:02,665-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [19cab49e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:00:02,668-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [19cab49e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:00:02,671-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19cab49e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 19cab49e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:00:02,674-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 142a3ea4
>2018-05-29 07:00:02,674-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:00:02,752-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2dba5295] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 26ab5baa
>2018-05-29 07:00:02,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2dba5295] FINISH, GetHardwareInfoVDSCommand, log id: 26ab5baa
>2018-05-29 07:00:02,766-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [702929f9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:00:02,775-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:00:02,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:00:02,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:00:02,775-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:00:02,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 142a3ea4
>2018-05-29 07:00:02,775-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:00:02,775-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [66e465b9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:00:02,781-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1af6c0a3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:00:02,787-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1af6c0a3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 29cf702
>2018-05-29 07:00:02,788-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [75489e3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:00:02,789-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [75489e3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1a6ddd7e
>2018-05-29 07:00:02,792-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [75489e3] FINISH, SetVdsStatusVDSCommand, log id: 1a6ddd7e
>2018-05-29 07:00:02,822-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [75489e3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 75489e3, Job ID: dffb8be6-5bfe-454b-aa44-ac82b4b3031c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:00:02,827-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [75489e3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2cd74724, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:00:02,829-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [75489e3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2cd74724, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:00:02,840-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [30596759] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:00:02,841-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [30596759] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:00:03,165-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1af6c0a3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:00:03,165-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1af6c0a3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 29cf702
>2018-05-29 07:00:03,169-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1af6c0a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:00:03,170-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1af6c0a3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:00:03,174-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1af6c0a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:00:03,175-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1af6c0a3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:00:03,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1af6c0a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:00:03,179-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1af6c0a3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:00:03,182-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1af6c0a3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:00:03,184-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1af6c0a3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1af6c0a3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:00:03,187-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 13be14b8
>2018-05-29 07:00:03,187-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:00:03,295-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:00:03,295-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:00:03,295-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:00:03,295-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:00:03,295-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 13be14b8
>2018-05-29 07:00:03,295-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:00:03,309-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [8b53a89] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:00:03,310-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [8b53a89] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 389c9c4b
>2018-05-29 07:00:03,312-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [8b53a89] FINISH, SetVdsStatusVDSCommand, log id: 389c9c4b
>2018-05-29 07:00:03,318-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8b53a89] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 8b53a89, Job ID: 86d3b5fc-985e-43c9-9de6-a8631f359583, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:00:03,323-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8b53a89] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 66e465b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:00:03,325-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8b53a89] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 66e465b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:00:03,336-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [48a4f5d1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:00:03,337-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [48a4f5d1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5c6287da] Autorecovering 2 hosts
>2018-05-29 07:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5c6287da] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:05:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [73ea879e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:05:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [73ea879e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:05:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [73ea879e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:05:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [73ea879e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:05:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [73ea879e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 37cfdcef
>2018-05-29 07:05:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [73ea879e] FINISH, SetVdsStatusVDSCommand, log id: 37cfdcef
>2018-05-29 07:05:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [73ea879e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:05:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [73ea879e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:05:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [73ea879e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:05:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [892938a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:05:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [892938a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:05:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [892938a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:05:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [892938a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:05:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [892938a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b8bb679
>2018-05-29 07:05:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [892938a] FINISH, SetVdsStatusVDSCommand, log id: b8bb679
>2018-05-29 07:05:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [892938a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1bf66285
>2018-05-29 07:05:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [892938a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1bf66285
>2018-05-29 07:05:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [892938a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:05:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [892938a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:05:00,486-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [1ebbfb73] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4b2e7a61
>2018-05-29 07:05:00,488-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [1ebbfb73] FINISH, GetHardwareInfoVDSCommand, log id: 4b2e7a61
>2018-05-29 07:05:00,501-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [7a1754ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:05:00,509-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [53796d10] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:05:00,514-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [48bcb69b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:05:00,526-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [48bcb69b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3a1a00e8
>2018-05-29 07:05:00,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [48bcb69b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:05:00,907-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [48bcb69b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3a1a00e8
>2018-05-29 07:05:00,909-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6cf6a443] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 56301026
>2018-05-29 07:05:00,911-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6cf6a443] FINISH, GetHardwareInfoVDSCommand, log id: 56301026
>2018-05-29 07:05:00,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [48bcb69b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:05:00,913-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [48bcb69b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:05:00,917-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [48bcb69b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:05:00,917-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [48bcb69b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:05:00,921-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [48bcb69b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:05:00,922-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [48bcb69b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:05:00,924-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [1b69d1fb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:05:00,926-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [48bcb69b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:05:00,928-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [6306895a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:05:00,929-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [48bcb69b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 48bcb69b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:05:00,933-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [12a0d5cf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:05:00,936-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7e3e4b2
>2018-05-29 07:05:00,936-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:05:00,946-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [12a0d5cf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2691e67a
>2018-05-29 07:05:01,033-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:05:01,033-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:05:01,034-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:05:01,034-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:05:01,034-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7e3e4b2
>2018-05-29 07:05:01,034-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:05:01,046-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [2e95eadf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:05:01,047-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2e95eadf] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5da3dc68
>2018-05-29 07:05:01,049-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2e95eadf] FINISH, SetVdsStatusVDSCommand, log id: 5da3dc68
>2018-05-29 07:05:01,057-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2e95eadf] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2e95eadf, Job ID: 24c20fc8-7899-4cb5-8728-671ed2dd2355, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:05:01,062-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2e95eadf] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 53796d10, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:05:01,064-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2e95eadf] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 53796d10, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:05:01,075-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5ab6961e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:05:01,076-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5ab6961e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:05:01,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12a0d5cf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:05:01,332-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [12a0d5cf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2691e67a
>2018-05-29 07:05:01,336-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12a0d5cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:05:01,337-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [12a0d5cf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:05:01,340-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12a0d5cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:05:01,341-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [12a0d5cf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:05:01,345-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12a0d5cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:05:01,345-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [12a0d5cf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:05:01,349-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [12a0d5cf] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:05:01,351-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12a0d5cf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 12a0d5cf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:05:01,358-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2a4496b4
>2018-05-29 07:05:01,358-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:05:01,459-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:05:01,459-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:05:01,459-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:05:01,459-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:05:01,459-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2a4496b4
>2018-05-29 07:05:01,459-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:05:01,472-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [2653bdc2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:05:01,474-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2653bdc2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 385261e5
>2018-05-29 07:05:01,476-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2653bdc2] FINISH, SetVdsStatusVDSCommand, log id: 385261e5
>2018-05-29 07:05:01,481-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2653bdc2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2653bdc2, Job ID: 5a1e62ea-7f04-4f1d-a6c5-efca81f00b15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:05:01,486-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2653bdc2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6306895a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:05:01,488-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2653bdc2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6306895a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:05:01,499-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [741fc064] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:05:01,500-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [741fc064] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:10:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5c07c8ef] Autorecovering 2 hosts
>2018-05-29 07:10:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5c07c8ef] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1bef4147] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:10:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1bef4147] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:10:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1bef4147] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:10:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1bef4147] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:10:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1bef4147] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 436fd524
>2018-05-29 07:10:00,081-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1bef4147] FINISH, SetVdsStatusVDSCommand, log id: 436fd524
>2018-05-29 07:10:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1bef4147] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:10:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1bef4147] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:10:00,085-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [1bef4147] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:10:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ec94dd1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:10:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ec94dd1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:10:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ec94dd1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:10:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ec94dd1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:10:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5ec94dd1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4e326b19
>2018-05-29 07:10:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5ec94dd1] FINISH, SetVdsStatusVDSCommand, log id: 4e326b19
>2018-05-29 07:10:00,103-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5ec94dd1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 72669b1b
>2018-05-29 07:10:00,105-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5ec94dd1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 72669b1b
>2018-05-29 07:10:00,105-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ec94dd1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:10:00,106-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ec94dd1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:10:01,769-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [537c2ec0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 794af37f
>2018-05-29 07:10:01,771-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [537c2ec0] FINISH, GetHardwareInfoVDSCommand, log id: 794af37f
>2018-05-29 07:10:01,785-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [7f633873] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:10:01,793-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [77f45fcc] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:10:01,798-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [661333d3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:10:01,805-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [661333d3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 17465723
>2018-05-29 07:10:02,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [277060d8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 770b6361
>2018-05-29 07:10:02,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [277060d8] FINISH, GetHardwareInfoVDSCommand, log id: 770b6361
>2018-05-29 07:10:02,112-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [60ad0554] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:10:02,116-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [7b9675aa] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:10:02,122-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [22759953] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:10:02,130-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [22759953] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5ba54580
>2018-05-29 07:10:02,174-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [661333d3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:10:02,174-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [661333d3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 17465723
>2018-05-29 07:10:02,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [661333d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:10:02,179-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [661333d3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:10:02,183-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [661333d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:10:02,184-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [661333d3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:10:02,188-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [661333d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:10:02,188-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [661333d3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:10:02,193-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [661333d3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:10:02,196-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [661333d3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 661333d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:10:02,199-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1bfdb66
>2018-05-29 07:10:02,199-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:10:02,297-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:10:02,297-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:10:02,297-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:10:02,297-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:10:02,297-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1bfdb66
>2018-05-29 07:10:02,297-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:10:02,312-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [6fdcf58d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:10:02,314-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6fdcf58d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2813ecd0
>2018-05-29 07:10:02,316-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6fdcf58d] FINISH, SetVdsStatusVDSCommand, log id: 2813ecd0
>2018-05-29 07:10:02,349-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6fdcf58d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6fdcf58d, Job ID: 831af585-870b-4ce9-86ed-f766fb50a5bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:10:02,354-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6fdcf58d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 77f45fcc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:10:02,357-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6fdcf58d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 77f45fcc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:10:02,368-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [76188f8e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:10:02,369-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [76188f8e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:10:02,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22759953] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:10:02,511-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [22759953] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5ba54580
>2018-05-29 07:10:02,515-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22759953] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:10:02,515-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [22759953] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:10:02,519-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22759953] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:10:02,520-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [22759953] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:10:02,524-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22759953] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:10:02,524-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [22759953] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:10:02,528-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [22759953] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:10:02,531-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22759953] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 22759953, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:10:02,534-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a2c6f61
>2018-05-29 07:10:02,534-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:10:02,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:10:02,633-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:10:02,633-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:10:02,633-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:10:02,633-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a2c6f61
>2018-05-29 07:10:02,633-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:10:02,647-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [42361695] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:10:02,648-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [42361695] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 133854d0
>2018-05-29 07:10:02,650-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [42361695] FINISH, SetVdsStatusVDSCommand, log id: 133854d0
>2018-05-29 07:10:02,656-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42361695] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 42361695, Job ID: 7620d78f-3668-4e9d-9e9e-7fb36ce7a80e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:10:02,660-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42361695] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7b9675aa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:10:02,662-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42361695] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7b9675aa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:10:02,673-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [23e28e2f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:10:02,674-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [23e28e2f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4e8bb5fd] Autorecovering 2 hosts
>2018-05-29 07:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4e8bb5fd] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:15:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [616431c6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [616431c6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [616431c6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [616431c6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:15:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [616431c6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8dcb6e8
>2018-05-29 07:15:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [616431c6] FINISH, SetVdsStatusVDSCommand, log id: 8dcb6e8
>2018-05-29 07:15:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [616431c6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:15:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [616431c6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:15:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [616431c6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:15:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a55c1ed] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:15:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a55c1ed] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:15:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a55c1ed] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:15:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a55c1ed] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:15:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6a55c1ed] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 60899ed4
>2018-05-29 07:15:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6a55c1ed] FINISH, SetVdsStatusVDSCommand, log id: 60899ed4
>2018-05-29 07:15:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [6a55c1ed] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: b0f7a7e
>2018-05-29 07:15:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [6a55c1ed] FINISH, SetHaMaintenanceModeVDSCommand, log id: b0f7a7e
>2018-05-29 07:15:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a55c1ed] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:15:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a55c1ed] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:15:00,268-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [492c37c2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1af4147c
>2018-05-29 07:15:00,270-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [492c37c2] FINISH, GetHardwareInfoVDSCommand, log id: 1af4147c
>2018-05-29 07:15:00,282-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [3f14d45f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:15:00,290-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [8488820] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:15:00,295-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [1a6f6f8f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:15:00,302-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [1a6f6f8f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: dc5c963
>2018-05-29 07:15:00,677-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1a6f6f8f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:15:00,677-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [1a6f6f8f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: dc5c963
>2018-05-29 07:15:00,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1a6f6f8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:15:00,682-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1a6f6f8f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:15:00,686-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1a6f6f8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:15:00,687-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1a6f6f8f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:15:00,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1a6f6f8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:15:00,691-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1a6f6f8f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:15:00,694-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [1a6f6f8f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:15:00,696-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1a6f6f8f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1a6f6f8f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:15:00,699-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 75e60f8d
>2018-05-29 07:15:00,699-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:15:00,798-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:15:00,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:15:00,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:15:00,798-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:15:00,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 75e60f8d
>2018-05-29 07:15:00,798-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:15:00,811-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [6fdfada9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:15:00,813-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6fdfada9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: cdf06f7
>2018-05-29 07:15:00,815-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6fdfada9] FINISH, SetVdsStatusVDSCommand, log id: cdf06f7
>2018-05-29 07:15:00,844-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6fdfada9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6fdfada9, Job ID: 473371bd-84f0-4eec-825a-ada20ee30c31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:15:00,848-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6fdfada9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 8488820, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:15:00,851-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6fdfada9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 8488820, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:15:00,862-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [316d39a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:15:00,863-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [316d39a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:15:03,873-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [49a52075] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 642a002a
>2018-05-29 07:15:03,875-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [49a52075] FINISH, GetHardwareInfoVDSCommand, log id: 642a002a
>2018-05-29 07:15:03,889-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [410abb6b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:15:03,898-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [33faf598] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:15:03,904-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4cd2a66e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:15:03,915-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4cd2a66e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: e915584
>2018-05-29 07:15:05,287-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4cd2a66e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:15:05,287-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4cd2a66e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: e915584
>2018-05-29 07:15:05,291-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4cd2a66e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:15:05,292-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4cd2a66e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:15:05,296-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4cd2a66e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:15:05,297-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4cd2a66e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:15:05,301-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4cd2a66e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:15:05,302-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4cd2a66e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:15:05,304-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4cd2a66e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:15:05,307-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4cd2a66e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4cd2a66e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:15:05,314-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 73b0a00f
>2018-05-29 07:15:05,314-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:15:05,413-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:15:05,413-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:15:05,413-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:15:05,413-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:15:05,413-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 73b0a00f
>2018-05-29 07:15:05,413-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:15:05,427-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [56c32cf5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:15:05,429-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [56c32cf5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: c6d4760
>2018-05-29 07:15:05,431-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [56c32cf5] FINISH, SetVdsStatusVDSCommand, log id: c6d4760
>2018-05-29 07:15:05,464-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [56c32cf5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 56c32cf5, Job ID: 6bd3e4ba-4b43-47ca-80cc-f92ab5d835ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:15:05,469-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [56c32cf5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 33faf598, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:15:05,471-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [56c32cf5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 33faf598, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:15:05,483-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [64241e8f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:15:05,484-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [64241e8f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [57a01d45] Autorecovering 2 hosts
>2018-05-29 07:20:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [57a01d45] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [60204a7b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:20:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [60204a7b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:20:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [60204a7b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:20:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [60204a7b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:20:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [60204a7b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b45b413
>2018-05-29 07:20:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [60204a7b] FINISH, SetVdsStatusVDSCommand, log id: 3b45b413
>2018-05-29 07:20:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [60204a7b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:20:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [60204a7b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:20:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [60204a7b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:20:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [308231df] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:20:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [308231df] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:20:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [308231df] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:20:00,094-04 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler2) [308231df] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS_INIT]', sharedLocks=''}'
>2018-05-29 07:20:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [308231df] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:20:00,106-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [308231df] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17d8adff
>2018-05-29 07:20:00,108-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [308231df] FINISH, SetVdsStatusVDSCommand, log id: 17d8adff
>2018-05-29 07:20:00,111-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [308231df] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3480c8df
>2018-05-29 07:20:00,113-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [308231df] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3480c8df
>2018-05-29 07:20:00,113-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [308231df] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:20:00,114-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [308231df] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:20:01,520-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [18ac7b8d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 69909e6a
>2018-05-29 07:20:01,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [18ac7b8d] FINISH, GetHardwareInfoVDSCommand, log id: 69909e6a
>2018-05-29 07:20:01,535-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6b476574] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:20:01,546-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [29857e25] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:20:01,551-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6ad84f30] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:20:01,564-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6ad84f30] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7981dcae
>2018-05-29 07:20:01,947-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6ad84f30] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:20:01,948-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6ad84f30] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7981dcae
>2018-05-29 07:20:01,951-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6ad84f30] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:20:01,952-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6ad84f30] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:20:01,956-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6ad84f30] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:20:01,957-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6ad84f30] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:20:01,961-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6ad84f30] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:20:01,961-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6ad84f30] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:20:01,965-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6ad84f30] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:20:01,968-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6ad84f30] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6ad84f30, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:20:01,976-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4c32a35a
>2018-05-29 07:20:01,976-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:20:02,075-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:20:02,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:20:02,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:20:02,075-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:20:02,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4c32a35a
>2018-05-29 07:20:02,075-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:20:02,090-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [2a267dd5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:20:02,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2a267dd5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7a40e21f
>2018-05-29 07:20:02,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2a267dd5] FINISH, SetVdsStatusVDSCommand, log id: 7a40e21f
>2018-05-29 07:20:02,125-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2a267dd5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2a267dd5, Job ID: d50e3048-49e4-4f11-bbe0-fb1b80076a36, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:20:02,130-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2a267dd5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29857e25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:20:02,132-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2a267dd5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29857e25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:20:02,144-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [534659a0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:20:02,145-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [534659a0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:20:03,235-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [c7d057a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 61f46cdf
>2018-05-29 07:20:03,236-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [c7d057a] FINISH, GetHardwareInfoVDSCommand, log id: 61f46cdf
>2018-05-29 07:20:03,250-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [38f074f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:20:03,258-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [7787e12a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:20:03,263-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4825892f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:20:03,275-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4825892f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 15a0460a
>2018-05-29 07:20:03,647-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4825892f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:20:03,647-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4825892f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 15a0460a
>2018-05-29 07:20:03,650-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4825892f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:20:03,651-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4825892f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:20:03,655-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4825892f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:20:03,655-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4825892f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:20:03,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4825892f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:20:03,660-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4825892f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:20:03,662-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4825892f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:20:03,665-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4825892f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4825892f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:20:03,671-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5aa93546
>2018-05-29 07:20:03,671-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:20:03,769-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:20:03,769-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:20:03,769-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:20:03,769-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:20:03,769-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5aa93546
>2018-05-29 07:20:03,769-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:20:03,781-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [5d62a5d3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:20:03,783-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5d62a5d3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f942773
>2018-05-29 07:20:03,784-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5d62a5d3] FINISH, SetVdsStatusVDSCommand, log id: 7f942773
>2018-05-29 07:20:03,816-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d62a5d3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5d62a5d3, Job ID: 38a4bd4e-9080-47c9-876f-cc5c7b5ebd16, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:20:03,821-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d62a5d3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7787e12a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:20:03,823-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d62a5d3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7787e12a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:20:03,833-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [49c3ae8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:20:03,834-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [49c3ae8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7a95c25d] Autorecovering 2 hosts
>2018-05-29 07:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7a95c25d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:25:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61caaae7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:25:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61caaae7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:25:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61caaae7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:25:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61caaae7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:25:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [61caaae7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4086c52
>2018-05-29 07:25:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [61caaae7] FINISH, SetVdsStatusVDSCommand, log id: 4086c52
>2018-05-29 07:25:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61caaae7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:25:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61caaae7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:25:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [61caaae7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:25:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7b92ae8f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:25:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7b92ae8f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:25:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7b92ae8f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:25:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7b92ae8f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:25:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7b92ae8f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7a0662ef
>2018-05-29 07:25:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7b92ae8f] FINISH, SetVdsStatusVDSCommand, log id: 7a0662ef
>2018-05-29 07:25:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7b92ae8f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 73150d2b
>2018-05-29 07:25:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7b92ae8f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 73150d2b
>2018-05-29 07:25:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7b92ae8f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:25:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7b92ae8f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:25:01,635-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [44c8fa9a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2ab307d9
>2018-05-29 07:25:01,637-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [44c8fa9a] FINISH, GetHardwareInfoVDSCommand, log id: 2ab307d9
>2018-05-29 07:25:01,651-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [52f5fcad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:25:01,660-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [66a24ed5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:25:01,665-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5d8c74da] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:25:01,671-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5d8c74da] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 31e51a01
>2018-05-29 07:25:02,039-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8c74da] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:25:02,039-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5d8c74da] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 31e51a01
>2018-05-29 07:25:02,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8c74da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:25:02,043-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5d8c74da] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:25:02,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8c74da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:25:02,048-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5d8c74da] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:25:02,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8c74da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:25:02,052-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5d8c74da] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:25:02,055-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5d8c74da] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:25:02,058-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8c74da] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5d8c74da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:25:02,061-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1f9d4ff8
>2018-05-29 07:25:02,061-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:25:02,160-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:25:02,160-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:25:02,160-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:25:02,160-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:25:02,160-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1f9d4ff8
>2018-05-29 07:25:02,160-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:25:02,174-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [555d1925] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:25:02,175-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [555d1925] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 374b066a
>2018-05-29 07:25:02,177-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [555d1925] FINISH, SetVdsStatusVDSCommand, log id: 374b066a
>2018-05-29 07:25:02,207-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [555d1925] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 555d1925, Job ID: c7995c02-020e-4a4d-bc6a-8645928617a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:25:02,212-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [555d1925] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 66a24ed5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:25:02,214-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [555d1925] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 66a24ed5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:25:02,225-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [64f4eaf0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:25:02,226-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [64f4eaf0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:25:02,865-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [71c7f937] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3dfd8e8c
>2018-05-29 07:25:02,867-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [71c7f937] FINISH, GetHardwareInfoVDSCommand, log id: 3dfd8e8c
>2018-05-29 07:25:02,880-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [4cd34317] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:25:02,884-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [6303f0d1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:25:02,890-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [442edd05] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:25:02,902-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [442edd05] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4210329f
>2018-05-29 07:25:03,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [442edd05] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:25:03,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [442edd05] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4210329f
>2018-05-29 07:25:03,284-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [442edd05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:25:03,284-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [442edd05] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:25:03,288-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [442edd05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:25:03,289-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [442edd05] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:25:03,293-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [442edd05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:25:03,294-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [442edd05] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:25:03,296-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [442edd05] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:25:03,299-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [442edd05] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 442edd05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:25:03,305-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6b18e531
>2018-05-29 07:25:03,305-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:25:03,402-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:25:03,402-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:25:03,402-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:25:03,402-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:25:03,402-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6b18e531
>2018-05-29 07:25:03,402-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:25:03,415-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [2c4f330] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:25:03,416-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2c4f330] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1195bf15
>2018-05-29 07:25:03,418-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2c4f330] FINISH, SetVdsStatusVDSCommand, log id: 1195bf15
>2018-05-29 07:25:03,453-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c4f330] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2c4f330, Job ID: 390922b1-0e29-4ebc-a0f1-6186c790538a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:25:03,457-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c4f330] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6303f0d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:25:03,459-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c4f330] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6303f0d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:25:03,470-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [2232abd5] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:25:03,471-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2232abd5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:30:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [354a3052] Autorecovering 2 hosts
>2018-05-29 07:30:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [354a3052] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:30:00,033-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [13d3c5c5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:30:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [13d3c5c5] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:30:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [13d3c5c5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:30:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [13d3c5c5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:30:00,041-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [13d3c5c5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4894dcd6
>2018-05-29 07:30:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [13d3c5c5] FINISH, SetVdsStatusVDSCommand, log id: 4894dcd6
>2018-05-29 07:30:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [13d3c5c5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:30:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [13d3c5c5] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:30:00,047-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [13d3c5c5] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:30:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [689277f6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:30:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [689277f6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:30:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [689277f6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:30:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [689277f6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:30:00,060-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [689277f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 70fee738
>2018-05-29 07:30:00,062-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [689277f6] FINISH, SetVdsStatusVDSCommand, log id: 70fee738
>2018-05-29 07:30:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [689277f6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5a77c610
>2018-05-29 07:30:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [689277f6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5a77c610
>2018-05-29 07:30:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [689277f6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:30:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [689277f6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:30:01,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [53cafe1e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2b3924a9
>2018-05-29 07:30:01,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [53cafe1e] FINISH, GetHardwareInfoVDSCommand, log id: 2b3924a9
>2018-05-29 07:30:01,085-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [10a43b3e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:30:01,090-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3f197cb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:30:01,095-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [181bae2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:30:01,107-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [181bae2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2d2725c1
>2018-05-29 07:30:01,485-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [181bae2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:30:01,485-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [181bae2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2d2725c1
>2018-05-29 07:30:01,489-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [181bae2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:30:01,489-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [181bae2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:30:01,493-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [181bae2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:30:01,494-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [181bae2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:30:01,498-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [181bae2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:30:01,499-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [181bae2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:30:01,501-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [181bae2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:30:01,504-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [181bae2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 181bae2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:30:01,511-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 56748444
>2018-05-29 07:30:01,511-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:30:01,611-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:30:01,611-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:30:01,611-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:30:01,611-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:30:01,611-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 56748444
>2018-05-29 07:30:01,611-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:30:01,625-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [a2a837] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:30:01,626-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [a2a837] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 775dcdd2
>2018-05-29 07:30:01,628-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [a2a837] FINISH, SetVdsStatusVDSCommand, log id: 775dcdd2
>2018-05-29 07:30:01,637-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [a2a837] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: a2a837, Job ID: 16cafe5a-5d45-48b6-94bd-527f5dbfebf4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:30:01,641-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [a2a837] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3f197cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:30:01,644-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [a2a837] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3f197cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:30:01,655-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [7aebe8e2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:30:01,656-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [7aebe8e2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:30:02,885-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [67c51276] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4289cce9
>2018-05-29 07:30:02,887-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [67c51276] FINISH, GetHardwareInfoVDSCommand, log id: 4289cce9
>2018-05-29 07:30:02,901-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [77fb800c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:30:02,905-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [434907d2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:30:02,911-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [6ae9345d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:30:02,918-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [6ae9345d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1df6902a
>2018-05-29 07:30:03,285-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ae9345d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:30:03,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [6ae9345d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1df6902a
>2018-05-29 07:30:03,289-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ae9345d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:30:03,290-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [6ae9345d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:30:03,294-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ae9345d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:30:03,295-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [6ae9345d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:30:03,298-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ae9345d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:30:03,299-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [6ae9345d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:30:03,348-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [6ae9345d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:30:03,352-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ae9345d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6ae9345d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:30:03,358-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: f2daaf0
>2018-05-29 07:30:03,358-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:30:03,460-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:30:03,460-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:30:03,460-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:30:03,460-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:30:03,460-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: f2daaf0
>2018-05-29 07:30:03,460-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:30:03,474-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [3b48282c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:30:03,476-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3b48282c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e408f6b
>2018-05-29 07:30:03,478-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3b48282c] FINISH, SetVdsStatusVDSCommand, log id: 5e408f6b
>2018-05-29 07:30:03,511-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b48282c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3b48282c, Job ID: e73e14a5-5f50-4f20-ab84-964145c9a570, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:30:03,516-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b48282c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 434907d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:30:03,518-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b48282c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 434907d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:30:03,529-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [6efc8832] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:30:03,530-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [6efc8832] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [688dca57] Autorecovering 2 hosts
>2018-05-29 07:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [688dca57] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:35:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [482150a6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:35:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [482150a6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:35:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [482150a6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:35:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [482150a6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:35:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [482150a6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4a749bd
>2018-05-29 07:35:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [482150a6] FINISH, SetVdsStatusVDSCommand, log id: 4a749bd
>2018-05-29 07:35:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [482150a6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:35:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [482150a6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:35:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [482150a6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:35:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [366ce5c0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:35:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [366ce5c0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:35:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [366ce5c0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:35:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [366ce5c0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:35:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [366ce5c0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12ce44e9
>2018-05-29 07:35:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [366ce5c0] FINISH, SetVdsStatusVDSCommand, log id: 12ce44e9
>2018-05-29 07:35:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [366ce5c0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6291685d
>2018-05-29 07:35:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [366ce5c0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6291685d
>2018-05-29 07:35:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [366ce5c0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:35:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [366ce5c0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:35:01,264-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [aa1d52] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: c787353
>2018-05-29 07:35:01,266-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [aa1d52] FINISH, GetHardwareInfoVDSCommand, log id: c787353
>2018-05-29 07:35:01,279-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3aac99c2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:35:01,288-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [26827007] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:35:01,293-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [233c2c27] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:35:01,305-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [233c2c27] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4b0d8601
>2018-05-29 07:35:01,678-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [233c2c27] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:35:01,678-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [233c2c27] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4b0d8601
>2018-05-29 07:35:01,683-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [233c2c27] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:35:01,683-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [233c2c27] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:35:01,687-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [233c2c27] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:35:01,688-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [233c2c27] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:35:01,692-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [233c2c27] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:35:01,692-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [233c2c27] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:35:01,695-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [233c2c27] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:35:01,698-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [233c2c27] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 233c2c27, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:35:01,704-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 569e0d52
>2018-05-29 07:35:01,704-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:35:01,799-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:35:01,799-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:35:01,799-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:35:01,799-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:35:01,799-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 569e0d52
>2018-05-29 07:35:01,799-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:35:01,813-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [39fa82f6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:35:01,814-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [39fa82f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4571374
>2018-05-29 07:35:01,816-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [39fa82f6] FINISH, SetVdsStatusVDSCommand, log id: 4571374
>2018-05-29 07:35:01,849-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39fa82f6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 39fa82f6, Job ID: 878b3fe2-2fc2-4cad-bb27-2ba88080232a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:35:01,853-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39fa82f6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 26827007, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:35:01,855-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39fa82f6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 26827007, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:35:01,867-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [8dde309] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:35:01,868-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [8dde309] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:35:02,286-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [1a875132] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 11d25337
>2018-05-29 07:35:02,287-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [1a875132] FINISH, GetHardwareInfoVDSCommand, log id: 11d25337
>2018-05-29 07:35:02,299-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [43a2967f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:35:02,304-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [1e0fb6f9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:35:02,309-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4f7892f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:35:02,316-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4f7892f8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5765ae90
>2018-05-29 07:35:02,691-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f7892f8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:35:02,691-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4f7892f8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5765ae90
>2018-05-29 07:35:02,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f7892f8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:35:02,695-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4f7892f8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:35:02,700-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f7892f8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:35:02,700-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4f7892f8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:35:02,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f7892f8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:35:02,705-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4f7892f8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:35:02,707-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4f7892f8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:35:02,710-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f7892f8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4f7892f8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:35:02,713-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 524cdd45
>2018-05-29 07:35:02,713-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:35:02,813-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:35:02,813-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:35:02,813-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:35:02,813-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:35:02,813-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 524cdd45
>2018-05-29 07:35:02,813-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:35:02,825-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [a48f9e3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:35:02,826-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [a48f9e3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 107ab3c
>2018-05-29 07:35:02,828-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [a48f9e3] FINISH, SetVdsStatusVDSCommand, log id: 107ab3c
>2018-05-29 07:35:02,833-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [a48f9e3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: a48f9e3, Job ID: b0abe400-a02c-49f9-bd11-f9e61a6887a7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:35:02,838-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [a48f9e3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1e0fb6f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:35:02,841-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [a48f9e3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1e0fb6f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:35:02,851-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [47a53365] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:35:02,853-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [47a53365] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [436fc54e] Autorecovering 2 hosts
>2018-05-29 07:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [436fc54e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:40:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2db2dcbd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2db2dcbd] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2db2dcbd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2db2dcbd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:40:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2db2dcbd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5d4e9ed5
>2018-05-29 07:40:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2db2dcbd] FINISH, SetVdsStatusVDSCommand, log id: 5d4e9ed5
>2018-05-29 07:40:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2db2dcbd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:40:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2db2dcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:40:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [2db2dcbd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:40:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5b2c45a5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:40:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5b2c45a5] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:40:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5b2c45a5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:40:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5b2c45a5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:40:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5b2c45a5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ae466e1
>2018-05-29 07:40:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5b2c45a5] FINISH, SetVdsStatusVDSCommand, log id: ae466e1
>2018-05-29 07:40:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [5b2c45a5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1cf460fb
>2018-05-29 07:40:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [5b2c45a5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1cf460fb
>2018-05-29 07:40:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5b2c45a5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:40:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5b2c45a5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:40:00,506-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6b2b3187] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2e2b4761
>2018-05-29 07:40:00,508-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6b2b3187] FINISH, GetHardwareInfoVDSCommand, log id: 2e2b4761
>2018-05-29 07:40:00,522-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [67d6be8d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:40:00,530-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [27a5811f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:40:00,535-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7fad7981] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:40:00,546-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7fad7981] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 17ae0170
>2018-05-29 07:40:00,927-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7fad7981] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:40:00,927-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7fad7981] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 17ae0170
>2018-05-29 07:40:00,931-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7fad7981] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:40:00,932-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7fad7981] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:40:00,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7fad7981] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:40:00,936-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7fad7981] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:40:00,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7fad7981] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:40:00,941-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7fad7981] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:40:00,944-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7fad7981] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:40:00,946-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7fad7981] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7fad7981, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:40:00,954-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: d6875f0
>2018-05-29 07:40:00,954-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:40:01,050-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:40:01,051-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:40:01,051-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:40:01,051-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:40:01,051-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: d6875f0
>2018-05-29 07:40:01,051-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:40:01,064-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [60e8ecf3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:40:01,065-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [60e8ecf3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42857756
>2018-05-29 07:40:01,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [60e8ecf3] FINISH, SetVdsStatusVDSCommand, log id: 42857756
>2018-05-29 07:40:01,100-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [60e8ecf3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 60e8ecf3, Job ID: b30bb010-1d8f-4e3f-b90a-722f7f8e7f60, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:40:01,105-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [60e8ecf3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 27a5811f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:40:01,107-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [60e8ecf3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 27a5811f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:40:01,118-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [4318c10] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:40:01,119-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [4318c10] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:40:02,622-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3499ddff] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6df770e2
>2018-05-29 07:40:02,624-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3499ddff] FINISH, GetHardwareInfoVDSCommand, log id: 6df770e2
>2018-05-29 07:40:02,636-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [35f85fa8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:40:02,641-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [2de3bded] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:40:02,646-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3242bae9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:40:02,653-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3242bae9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1af671fa
>2018-05-29 07:40:03,023-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3242bae9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:40:03,023-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3242bae9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1af671fa
>2018-05-29 07:40:03,027-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3242bae9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:40:03,027-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3242bae9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:40:03,031-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3242bae9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:40:03,032-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3242bae9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:40:03,036-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3242bae9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:40:03,037-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3242bae9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:40:03,040-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3242bae9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:40:03,043-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3242bae9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3242bae9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:40:03,050-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6e92680a
>2018-05-29 07:40:03,050-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:40:03,148-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:40:03,148-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:40:03,148-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:40:03,148-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:40:03,148-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6e92680a
>2018-05-29 07:40:03,149-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:40:03,163-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5ad07023] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:40:03,164-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5ad07023] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 316b0384
>2018-05-29 07:40:03,166-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5ad07023] FINISH, SetVdsStatusVDSCommand, log id: 316b0384
>2018-05-29 07:40:03,197-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ad07023] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5ad07023, Job ID: 2e89fc54-0ca8-40a5-881f-fbee7afe062b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:40:03,202-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ad07023] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2de3bded, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:40:03,204-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ad07023] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2de3bded, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:40:03,215-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [3be5efe3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:40:03,216-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [3be5efe3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [f61b467] Autorecovering 2 hosts
>2018-05-29 07:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [f61b467] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:45:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2e633e15] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2e633e15] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2e633e15] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2e633e15] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:45:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2e633e15] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1d7beaff
>2018-05-29 07:45:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2e633e15] FINISH, SetVdsStatusVDSCommand, log id: 1d7beaff
>2018-05-29 07:45:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2e633e15] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:45:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2e633e15] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:45:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [2e633e15] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:45:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [3b2c60cb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:45:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [3b2c60cb] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:45:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [3b2c60cb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:45:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [3b2c60cb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:45:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3b2c60cb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 208929f2
>2018-05-29 07:45:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3b2c60cb] FINISH, SetVdsStatusVDSCommand, log id: 208929f2
>2018-05-29 07:45:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [3b2c60cb] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 539e5e45
>2018-05-29 07:45:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [3b2c60cb] FINISH, SetHaMaintenanceModeVDSCommand, log id: 539e5e45
>2018-05-29 07:45:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [3b2c60cb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:45:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [3b2c60cb] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:45:00,878-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [338949cb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1e350716
>2018-05-29 07:45:00,880-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [338949cb] FINISH, GetHardwareInfoVDSCommand, log id: 1e350716
>2018-05-29 07:45:00,893-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [32495681] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:45:00,901-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [1a2d3308] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:45:00,906-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [70f3b6bf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:45:00,964-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [70f3b6bf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3e6dfdd3
>2018-05-29 07:45:01,335-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70f3b6bf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:45:01,335-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [70f3b6bf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3e6dfdd3
>2018-05-29 07:45:01,339-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70f3b6bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:45:01,340-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [70f3b6bf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:45:01,343-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70f3b6bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:45:01,344-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [70f3b6bf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:45:01,347-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70f3b6bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:45:01,348-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [70f3b6bf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:45:01,351-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [70f3b6bf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:45:01,353-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70f3b6bf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 70f3b6bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:45:01,360-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 173327e5
>2018-05-29 07:45:01,360-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:45:01,457-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:45:01,457-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:45:01,457-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:45:01,457-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:45:01,457-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 173327e5
>2018-05-29 07:45:01,457-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:45:01,470-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [1ff6be63] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:45:01,471-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1ff6be63] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3245f397
>2018-05-29 07:45:01,474-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1ff6be63] FINISH, SetVdsStatusVDSCommand, log id: 3245f397
>2018-05-29 07:45:01,482-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ff6be63] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1ff6be63, Job ID: 3314a4c2-023e-43d0-9404-7bbb3006caad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:45:01,486-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ff6be63] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1a2d3308, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:45:01,489-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ff6be63] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1a2d3308, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:45:01,499-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [78dd2f2d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:45:01,500-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [78dd2f2d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:45:01,703-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4d0ccadf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 420a658
>2018-05-29 07:45:01,704-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4d0ccadf] FINISH, GetHardwareInfoVDSCommand, log id: 420a658
>2018-05-29 07:45:01,716-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [72b5c834] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:45:01,721-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3b5d337a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:45:01,726-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [1514d148] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:45:01,732-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [1514d148] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e1af551
>2018-05-29 07:45:02,104-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1514d148] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:45:02,104-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [1514d148] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e1af551
>2018-05-29 07:45:02,108-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1514d148] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:45:02,109-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1514d148] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:45:02,113-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1514d148] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:45:02,114-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1514d148] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:45:02,117-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1514d148] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:45:02,118-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1514d148] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:45:02,121-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [1514d148] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:45:02,124-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1514d148] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1514d148, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:45:02,127-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: e9c8834
>2018-05-29 07:45:02,127-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:45:02,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:45:02,226-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:45:02,226-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:45:02,226-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:45:02,226-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: e9c8834
>2018-05-29 07:45:02,226-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:45:02,238-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [586c3682] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:45:02,240-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [586c3682] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 34fefa3c
>2018-05-29 07:45:02,242-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [586c3682] FINISH, SetVdsStatusVDSCommand, log id: 34fefa3c
>2018-05-29 07:45:02,247-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [586c3682] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 586c3682, Job ID: 33e05068-2433-4679-adff-4ae39202cb3b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:45:02,252-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [586c3682] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b5d337a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:45:02,254-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [586c3682] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b5d337a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:45:02,265-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [41424396] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:45:02,266-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [41424396] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [536c7eb3] Autorecovering 2 hosts
>2018-05-29 07:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [536c7eb3] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:50:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [741d8e4a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:50:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [741d8e4a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:50:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [741d8e4a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:50:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [741d8e4a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:50:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [741d8e4a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53ed4116
>2018-05-29 07:50:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [741d8e4a] FINISH, SetVdsStatusVDSCommand, log id: 53ed4116
>2018-05-29 07:50:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [741d8e4a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:50:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [741d8e4a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:50:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [741d8e4a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:50:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7e1f9d08] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:50:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7e1f9d08] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:50:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7e1f9d08] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:50:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7e1f9d08] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:50:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7e1f9d08] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f292117
>2018-05-29 07:50:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7e1f9d08] FINISH, SetVdsStatusVDSCommand, log id: 5f292117
>2018-05-29 07:50:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [7e1f9d08] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2da1be0c
>2018-05-29 07:50:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [7e1f9d08] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2da1be0c
>2018-05-29 07:50:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7e1f9d08] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:50:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7e1f9d08] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:50:02,130-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [129c2af8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 30c684e7
>2018-05-29 07:50:02,132-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [129c2af8] FINISH, GetHardwareInfoVDSCommand, log id: 30c684e7
>2018-05-29 07:50:02,146-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [6a2dc695] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:50:02,154-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [617adc09] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:50:02,159-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5eaa84d8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:50:02,171-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5eaa84d8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 8cb259e
>2018-05-29 07:50:02,539-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eaa84d8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:50:02,539-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5eaa84d8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 8cb259e
>2018-05-29 07:50:02,547-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eaa84d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:50:02,548-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5eaa84d8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:50:02,551-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eaa84d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:50:02,552-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5eaa84d8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:50:02,556-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eaa84d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:50:02,556-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5eaa84d8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:50:02,559-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5eaa84d8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:50:02,562-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eaa84d8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5eaa84d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:50:02,568-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 60662bf5
>2018-05-29 07:50:02,568-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:50:02,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:50:02,664-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:50:02,664-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:50:02,664-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:50:02,664-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 60662bf5
>2018-05-29 07:50:02,664-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:50:02,678-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [39d9df17] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:50:02,679-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [39d9df17] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76de70
>2018-05-29 07:50:02,681-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [39d9df17] FINISH, SetVdsStatusVDSCommand, log id: 76de70
>2018-05-29 07:50:02,712-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [39d9df17] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 39d9df17, Job ID: 85f9f564-ab70-454d-a297-c7c1fba4ff84, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:50:02,717-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [39d9df17] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 617adc09, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:50:02,719-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [39d9df17] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 617adc09, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:50:02,729-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [5ec568a3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:50:02,730-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [5ec568a3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:50:03,169-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [67fda786] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3a67eec2
>2018-05-29 07:50:03,171-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [67fda786] FINISH, GetHardwareInfoVDSCommand, log id: 3a67eec2
>2018-05-29 07:50:03,184-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [442a2317] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:50:03,192-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [658f7146] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:50:03,197-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [515274e9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:50:03,209-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [515274e9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c3a72ae
>2018-05-29 07:50:03,598-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [515274e9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:50:03,598-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [515274e9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c3a72ae
>2018-05-29 07:50:03,602-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [515274e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:50:03,603-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [515274e9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:50:03,607-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [515274e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:50:03,608-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [515274e9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:50:03,611-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [515274e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:50:03,612-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [515274e9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:50:03,615-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [515274e9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:50:03,617-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [515274e9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 515274e9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:50:03,624-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2e6f5793
>2018-05-29 07:50:03,624-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:50:03,723-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:50:03,723-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:50:03,723-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:50:03,723-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:50:03,723-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2e6f5793
>2018-05-29 07:50:03,723-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:50:03,735-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [65b7046d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:50:03,736-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [65b7046d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4875b673
>2018-05-29 07:50:03,738-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [65b7046d] FINISH, SetVdsStatusVDSCommand, log id: 4875b673
>2018-05-29 07:50:03,769-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65b7046d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 65b7046d, Job ID: a31adbc8-a0b4-4531-950e-51ae8b7efb87, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:50:03,774-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65b7046d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 658f7146, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:50:03,776-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65b7046d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 658f7146, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:50:03,787-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [8649add] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:50:03,788-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [8649add] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:54:38,333-04 INFO  [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler7) [469b59d4] Backup check started.
>2018-05-29 07:54:38,349-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [469b59d4] EVENT_ID: ENGINE_NO_FULL_BACKUP(9,022), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: There is no full backup available, please run engine-backup to prevent data loss in case of corruption.
>2018-05-29 07:54:38,349-04 INFO  [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler7) [469b59d4] Backup check completed.
>2018-05-29 07:54:42,906-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler7) [5dc7eaba] Attempting to update VMs/Templates Ovf.
>2018-05-29 07:54:42,909-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 07:54:42,909-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 07:54:42,910-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:54:42,960-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 07:54:42,964-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 07:54:42,964-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 07:54:42,964-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 07:54:42,964-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 07:54:42,965-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 07:54:42,966-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 07:55:00,014-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [716dbdf8] Autorecovering 2 hosts
>2018-05-29 07:55:00,014-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [716dbdf8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d2b9edd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:55:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d2b9edd] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:55:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d2b9edd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:55:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d2b9edd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:55:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1d2b9edd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c270978
>2018-05-29 07:55:00,080-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1d2b9edd] FINISH, SetVdsStatusVDSCommand, log id: 2c270978
>2018-05-29 07:55:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d2b9edd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:55:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d2b9edd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 07:55:00,082-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1d2b9edd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:55:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [63b94db0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:55:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [63b94db0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 07:55:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [63b94db0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:55:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [63b94db0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:55:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [63b94db0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 9b2f132
>2018-05-29 07:55:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [63b94db0] FINISH, SetVdsStatusVDSCommand, log id: 9b2f132
>2018-05-29 07:55:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [63b94db0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6caeaffe
>2018-05-29 07:55:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [63b94db0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6caeaffe
>2018-05-29 07:55:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [63b94db0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 07:55:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [63b94db0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 07:55:00,391-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [74887fad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 15b4ec6e
>2018-05-29 07:55:00,393-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [74887fad] FINISH, GetHardwareInfoVDSCommand, log id: 15b4ec6e
>2018-05-29 07:55:00,405-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [5e7c96bb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:55:00,410-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [730ea822] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:55:00,415-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [60e5861d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:55:00,426-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [60e5861d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c6a8ae0
>2018-05-29 07:55:00,813-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [60e5861d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:55:00,813-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [60e5861d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c6a8ae0
>2018-05-29 07:55:00,817-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [60e5861d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:55:00,817-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [60e5861d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:55:00,821-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [60e5861d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:55:00,822-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [60e5861d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:55:00,826-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [60e5861d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:55:00,827-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [60e5861d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:55:00,830-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [60e5861d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:55:00,832-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [60e5861d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 60e5861d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:55:00,839-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 676552ce
>2018-05-29 07:55:00,839-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:55:00,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:55:00,938-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:55:00,938-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:55:00,938-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:55:00,938-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 676552ce
>2018-05-29 07:55:00,938-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:55:00,952-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4f98df48] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:55:00,954-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4f98df48] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a829703
>2018-05-29 07:55:00,956-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4f98df48] FINISH, SetVdsStatusVDSCommand, log id: 5a829703
>2018-05-29 07:55:00,964-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f98df48] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4f98df48, Job ID: 14d40912-6dbd-4b96-b620-af63f7765b0e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:55:00,969-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f98df48] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 730ea822, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:55:00,971-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f98df48] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 730ea822, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:55:00,982-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [312353c0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 07:55:00,983-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [312353c0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 07:55:01,408-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5fb70c89] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7c51ee27
>2018-05-29 07:55:01,410-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5fb70c89] FINISH, GetHardwareInfoVDSCommand, log id: 7c51ee27
>2018-05-29 07:55:01,422-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [79c4e81c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:55:01,427-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [48536b05] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:55:01,432-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7e6c8610] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 07:55:01,438-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7e6c8610] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 304c5f88
>2018-05-29 07:55:01,815-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e6c8610] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 07:55:01,815-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7e6c8610] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 304c5f88
>2018-05-29 07:55:01,819-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e6c8610] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:55:01,819-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7e6c8610] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:55:01,823-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e6c8610] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:55:01,824-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7e6c8610] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:55:01,828-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e6c8610] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 07:55:01,829-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7e6c8610] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 07:55:01,831-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7e6c8610] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 07:55:01,834-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e6c8610] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7e6c8610, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 07:55:01,838-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5b2e8b74
>2018-05-29 07:55:01,838-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 07:55:01,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:55:01,938-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 07:55:01,938-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 07:55:01,938-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 07:55:01,938-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5b2e8b74
>2018-05-29 07:55:01,938-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 07:55:01,952-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [110c450] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:55:01,953-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [110c450] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3442b3e8
>2018-05-29 07:55:01,956-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [110c450] FINISH, SetVdsStatusVDSCommand, log id: 3442b3e8
>2018-05-29 07:55:01,961-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [110c450] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 110c450, Job ID: 9b71602c-515f-4ca2-86d1-90213ec6f941, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 07:55:01,966-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [110c450] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 48536b05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 07:55:01,968-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [110c450] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 48536b05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 07:55:01,980-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [18eb1ce3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 07:55:01,981-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [18eb1ce3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [ab9a723] Autorecovering 2 hosts
>2018-05-29 08:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [ab9a723] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:00:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [510ca8b4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [510ca8b4] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [510ca8b4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [510ca8b4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:00:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [510ca8b4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 477d7e37
>2018-05-29 08:00:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [510ca8b4] FINISH, SetVdsStatusVDSCommand, log id: 477d7e37
>2018-05-29 08:00:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [510ca8b4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:00:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [510ca8b4] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:00:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [510ca8b4] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:00:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [435c937c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [435c937c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [435c937c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [435c937c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:00:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [435c937c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31edd54
>2018-05-29 08:00:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [435c937c] FINISH, SetVdsStatusVDSCommand, log id: 31edd54
>2018-05-29 08:00:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [435c937c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 58848b14
>2018-05-29 08:00:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [435c937c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 58848b14
>2018-05-29 08:00:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [435c937c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:00:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [435c937c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:00:01,629-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [c4576e6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 24939a5a
>2018-05-29 08:00:01,630-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [c4576e6] FINISH, GetHardwareInfoVDSCommand, log id: 24939a5a
>2018-05-29 08:00:01,643-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [75102a3c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:00:01,651-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [5e980b66] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:00:01,656-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [1ccb8db7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:00:01,668-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [1ccb8db7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4deb458f
>2018-05-29 08:00:02,038-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ccb8db7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:00:02,038-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [1ccb8db7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4deb458f
>2018-05-29 08:00:02,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ccb8db7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:00:02,042-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [1ccb8db7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:00:02,046-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ccb8db7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:00:02,047-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [1ccb8db7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:00:02,051-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ccb8db7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:00:02,051-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [1ccb8db7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:00:02,054-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [1ccb8db7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:00:02,057-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ccb8db7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1ccb8db7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:00:02,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 58ff4e33
>2018-05-29 08:00:02,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:00:02,159-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:00:02,159-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:00:02,159-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:00:02,159-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:00:02,159-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 58ff4e33
>2018-05-29 08:00:02,159-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:00:02,172-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [31f5a989] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:00:02,173-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [31f5a989] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6a0e17a
>2018-05-29 08:00:02,175-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [31f5a989] FINISH, SetVdsStatusVDSCommand, log id: 6a0e17a
>2018-05-29 08:00:02,207-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [31f5a989] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 31f5a989, Job ID: e333a530-e8f6-4b1c-8acc-f32f1262251d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:00:02,211-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [31f5a989] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5e980b66, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:00:02,213-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [31f5a989] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5e980b66, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:00:02,224-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2840a4c0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:00:02,225-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2840a4c0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:00:02,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [68951fb1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 47d7e11b
>2018-05-29 08:00:02,612-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [68951fb1] FINISH, GetHardwareInfoVDSCommand, log id: 47d7e11b
>2018-05-29 08:00:02,624-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [5ffc0f50] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:00:02,629-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [15650649] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:00:02,634-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [23b47465] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:00:02,640-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [23b47465] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 44c15250
>2018-05-29 08:00:03,013-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23b47465] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:00:03,013-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [23b47465] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 44c15250
>2018-05-29 08:00:03,016-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23b47465] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:00:03,017-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [23b47465] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:00:03,021-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23b47465] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:00:03,022-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [23b47465] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:00:03,025-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23b47465] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:00:03,026-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [23b47465] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:00:03,029-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [23b47465] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:00:03,031-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23b47465] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 23b47465, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:00:03,034-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 675112f1
>2018-05-29 08:00:03,034-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:00:03,132-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:00:03,132-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:00:03,132-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:00:03,132-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:00:03,132-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 675112f1
>2018-05-29 08:00:03,132-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:00:03,145-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [7b25b152] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:00:03,146-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7b25b152] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 115eaad0
>2018-05-29 08:00:03,148-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7b25b152] FINISH, SetVdsStatusVDSCommand, log id: 115eaad0
>2018-05-29 08:00:03,154-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7b25b152] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7b25b152, Job ID: 2c7a30f0-a43a-4324-91d6-620ef69eee9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:00:03,158-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7b25b152] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 15650649, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:00:03,160-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7b25b152] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 15650649, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:00:03,170-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [459c10cd] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:00:03,171-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [459c10cd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6cd5b6d2] Autorecovering 2 hosts
>2018-05-29 08:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6cd5b6d2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:05:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [248de92c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:05:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [248de92c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:05:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [248de92c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:05:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [248de92c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:05:00,049-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [248de92c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4eb14dd9
>2018-05-29 08:05:00,052-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [248de92c] FINISH, SetVdsStatusVDSCommand, log id: 4eb14dd9
>2018-05-29 08:05:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [248de92c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:05:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [248de92c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:05:00,055-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [248de92c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:05:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6a95de9e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:05:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6a95de9e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:05:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6a95de9e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:05:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6a95de9e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:05:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6a95de9e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 615bcde2
>2018-05-29 08:05:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6a95de9e] FINISH, SetVdsStatusVDSCommand, log id: 615bcde2
>2018-05-29 08:05:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [6a95de9e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 21dae6f0
>2018-05-29 08:05:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [6a95de9e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 21dae6f0
>2018-05-29 08:05:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6a95de9e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:05:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6a95de9e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:05:00,803-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2fc9ef36] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7e1f464d
>2018-05-29 08:05:00,805-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2fc9ef36] FINISH, GetHardwareInfoVDSCommand, log id: 7e1f464d
>2018-05-29 08:05:00,817-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [5f76b131] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:05:00,826-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [20c3fb7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:05:00,831-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [583083bf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:05:00,842-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [583083bf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5a322c80
>2018-05-29 08:05:01,229-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [583083bf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:05:01,229-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [583083bf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5a322c80
>2018-05-29 08:05:01,233-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [583083bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:05:01,234-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [583083bf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:05:01,238-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [583083bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:05:01,239-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [583083bf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:05:01,243-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [583083bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:05:01,243-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [583083bf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:05:01,246-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [583083bf] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:05:01,249-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [583083bf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 583083bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:05:01,255-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3013cfde
>2018-05-29 08:05:01,255-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:05:01,354-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-33) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:05:01,354-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:05:01,354-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:05:01,354-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:05:01,355-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3013cfde
>2018-05-29 08:05:01,355-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-33) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:05:01,368-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [587151f6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:05:01,369-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [587151f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 408c0eff
>2018-05-29 08:05:01,371-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [587151f6] FINISH, SetVdsStatusVDSCommand, log id: 408c0eff
>2018-05-29 08:05:01,402-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [587151f6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 587151f6, Job ID: 38ef46f2-2e69-4959-a88a-d38a900c7c72, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:05:01,407-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [587151f6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 20c3fb7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:05:01,409-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [587151f6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 20c3fb7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:05:01,419-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7f521be2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:05:01,420-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7f521be2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:05:02,908-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [9ad4c5e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7ea19ea
>2018-05-29 08:05:02,911-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [9ad4c5e] FINISH, GetHardwareInfoVDSCommand, log id: 7ea19ea
>2018-05-29 08:05:02,924-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [5d2e6790] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:05:02,929-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [29918fb8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:05:02,934-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [44d9d918] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:05:02,941-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [44d9d918] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3dc1270a
>2018-05-29 08:05:03,314-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44d9d918] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:05:03,314-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [44d9d918] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3dc1270a
>2018-05-29 08:05:03,318-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44d9d918] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:05:03,319-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [44d9d918] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:05:03,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44d9d918] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:05:03,323-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [44d9d918] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:05:03,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44d9d918] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:05:03,328-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [44d9d918] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:05:03,331-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [44d9d918] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:05:03,334-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44d9d918] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 44d9d918, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:05:03,337-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 76916ebc
>2018-05-29 08:05:03,337-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:05:03,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:05:03,436-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:05:03,436-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:05:03,436-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:05:03,436-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 76916ebc
>2018-05-29 08:05:03,436-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:05:03,450-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [1d308030] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:05:03,451-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1d308030] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3725152
>2018-05-29 08:05:03,453-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1d308030] FINISH, SetVdsStatusVDSCommand, log id: 3725152
>2018-05-29 08:05:03,484-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d308030] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1d308030, Job ID: da218389-8855-46ae-a8d0-dc5a79d4e68d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:05:03,488-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d308030] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29918fb8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:05:03,491-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d308030] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29918fb8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:05:03,502-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [757991a4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:05:03,503-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [757991a4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [57bbf925] Autorecovering 2 hosts
>2018-05-29 08:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [57bbf925] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:10:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4cb179b7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4cb179b7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4cb179b7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4cb179b7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:10:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4cb179b7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1ee0fc94
>2018-05-29 08:10:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4cb179b7] FINISH, SetVdsStatusVDSCommand, log id: 1ee0fc94
>2018-05-29 08:10:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4cb179b7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:10:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4cb179b7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:10:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [4cb179b7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:10:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50f67730] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50f67730] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50f67730] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50f67730] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:10:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [50f67730] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b9e1ded
>2018-05-29 08:10:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [50f67730] FINISH, SetVdsStatusVDSCommand, log id: b9e1ded
>2018-05-29 08:10:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [50f67730] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 51304bef
>2018-05-29 08:10:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [50f67730] FINISH, SetHaMaintenanceModeVDSCommand, log id: 51304bef
>2018-05-29 08:10:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50f67730] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:10:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50f67730] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:10:01,214-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3304b684] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: d14e98d
>2018-05-29 08:10:01,216-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3304b684] FINISH, GetHardwareInfoVDSCommand, log id: d14e98d
>2018-05-29 08:10:01,230-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [ae2f096] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:10:01,239-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [552590d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:10:01,244-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5ce86f73] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:10:01,255-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5ce86f73] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6a101daf
>2018-05-29 08:10:02,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [2b3dfe35] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 512aab9d
>2018-05-29 08:10:02,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [2b3dfe35] FINISH, GetHardwareInfoVDSCommand, log id: 512aab9d
>2018-05-29 08:10:02,106-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [2d97499a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:10:02,110-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [5098c04d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:10:02,115-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4d0734cd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:10:02,122-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4d0734cd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 30a4f4a1
>2018-05-29 08:10:02,503-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4d0734cd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:10:02,503-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4d0734cd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 30a4f4a1
>2018-05-29 08:10:02,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4d0734cd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:10:02,508-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4d0734cd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:10:02,512-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4d0734cd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:10:02,513-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4d0734cd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:10:02,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4d0734cd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:10:02,517-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4d0734cd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:10:02,521-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4d0734cd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:10:02,523-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4d0734cd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d0734cd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:10:02,530-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 57bda063
>2018-05-29 08:10:02,530-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:10:02,618-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5ce86f73] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:10:02,618-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5ce86f73] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6a101daf
>2018-05-29 08:10:02,622-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5ce86f73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:10:02,622-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5ce86f73] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:10:02,626-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5ce86f73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:10:02,627-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5ce86f73] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:10:02,628-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:10:02,628-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:10:02,628-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:10:02,628-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:10:02,628-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 57bda063
>2018-05-29 08:10:02,628-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:10:02,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5ce86f73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:10:02,632-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5ce86f73] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:10:02,636-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5ce86f73] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:10:02,639-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5ce86f73] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5ce86f73, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:10:02,641-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [1eee2ba3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:10:02,642-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 388daeab
>2018-05-29 08:10:02,642-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:10:02,642-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1eee2ba3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e54788c
>2018-05-29 08:10:02,645-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1eee2ba3] FINISH, SetVdsStatusVDSCommand, log id: 5e54788c
>2018-05-29 08:10:02,677-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1eee2ba3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1eee2ba3, Job ID: 74adbddf-dacb-48f5-aac0-702e102527ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:10:02,682-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1eee2ba3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5098c04d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:10:02,684-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1eee2ba3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5098c04d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:10:02,695-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [4684d7d8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:10:02,696-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [4684d7d8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:10:02,742-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:10:02,742-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:10:02,742-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:10:02,742-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:10:02,742-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 388daeab
>2018-05-29 08:10:02,742-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:10:02,754-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [72c264e2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:10:02,755-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [72c264e2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 11d4d6
>2018-05-29 08:10:02,757-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [72c264e2] FINISH, SetVdsStatusVDSCommand, log id: 11d4d6
>2018-05-29 08:10:02,762-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [72c264e2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 72c264e2, Job ID: dc80abd9-a0dd-4523-8dd6-ae3789566ba6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:10:02,767-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [72c264e2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 552590d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:10:02,769-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [72c264e2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 552590d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:10:02,779-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [1e905ec8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:10:02,780-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [1e905ec8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [25cf493e] Autorecovering 2 hosts
>2018-05-29 08:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [25cf493e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:15:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7bc6240c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7bc6240c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7bc6240c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7bc6240c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:15:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7bc6240c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6cfc70ac
>2018-05-29 08:15:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7bc6240c] FINISH, SetVdsStatusVDSCommand, log id: 6cfc70ac
>2018-05-29 08:15:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7bc6240c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:15:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7bc6240c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:15:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7bc6240c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:15:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22cff3b6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:15:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22cff3b6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:15:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22cff3b6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:15:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22cff3b6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:15:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [22cff3b6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19c4a6cf
>2018-05-29 08:15:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [22cff3b6] FINISH, SetVdsStatusVDSCommand, log id: 19c4a6cf
>2018-05-29 08:15:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [22cff3b6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 765280a1
>2018-05-29 08:15:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [22cff3b6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 765280a1
>2018-05-29 08:15:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22cff3b6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:15:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22cff3b6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:15:00,387-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1045ea68] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5d02b4e9
>2018-05-29 08:15:00,389-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1045ea68] FINISH, GetHardwareInfoVDSCommand, log id: 5d02b4e9
>2018-05-29 08:15:00,402-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7509f81a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:15:00,411-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [99edcf8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:15:00,417-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [77ed9772] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:15:00,481-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [77ed9772] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3f4baaa9
>2018-05-29 08:15:00,536-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [66dc9526] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 78b81222
>2018-05-29 08:15:00,537-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [66dc9526] FINISH, GetHardwareInfoVDSCommand, log id: 78b81222
>2018-05-29 08:15:00,551-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [13e68c51] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:15:00,555-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [32979254] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:15:00,560-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [6871fc02] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:15:00,567-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [6871fc02] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: e7bc0eb
>2018-05-29 08:15:00,861-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77ed9772] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:15:00,861-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [77ed9772] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3f4baaa9
>2018-05-29 08:15:00,865-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77ed9772] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:15:00,865-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [77ed9772] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:15:00,869-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77ed9772] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:15:00,870-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [77ed9772] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:15:00,874-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77ed9772] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:15:00,874-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [77ed9772] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:15:00,877-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [77ed9772] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:15:00,880-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77ed9772] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 77ed9772, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:15:00,887-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2fd7b5cf
>2018-05-29 08:15:00,887-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:15:00,937-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6871fc02] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:15:00,937-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [6871fc02] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: e7bc0eb
>2018-05-29 08:15:00,941-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6871fc02] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:15:00,942-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [6871fc02] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:15:00,946-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6871fc02] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:15:00,946-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [6871fc02] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:15:00,950-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6871fc02] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:15:00,951-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [6871fc02] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:15:00,953-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [6871fc02] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:15:00,956-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6871fc02] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6871fc02, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:15:00,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:15:00,987-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:15:00,987-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:15:00,987-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:15:00,987-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2fd7b5cf
>2018-05-29 08:15:00,987-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:15:00,988-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1fef81f7
>2018-05-29 08:15:00,988-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:15:01,000-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [5f5491b7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:15:01,002-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5f5491b7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f8e1e9
>2018-05-29 08:15:01,004-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5f5491b7] FINISH, SetVdsStatusVDSCommand, log id: f8e1e9
>2018-05-29 08:15:01,035-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5f5491b7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5f5491b7, Job ID: dc39eba3-5707-4e34-8500-eb8bc6897d3c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:15:01,040-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5f5491b7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 99edcf8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:15:01,042-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5f5491b7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 99edcf8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:15:01,053-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [5f2cb41b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:15:01,054-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [5f2cb41b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:15:01,087-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:15:01,087-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:15:01,087-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:15:01,087-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:15:01,087-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1fef81f7
>2018-05-29 08:15:01,087-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:15:01,101-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [226d75] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:15:01,102-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [226d75] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7e49d1f2
>2018-05-29 08:15:01,104-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [226d75] FINISH, SetVdsStatusVDSCommand, log id: 7e49d1f2
>2018-05-29 08:15:01,110-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [226d75] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 226d75, Job ID: 1e7ab2f3-8959-4882-8f5b-4a8786a86d1b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:15:01,114-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [226d75] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 32979254, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:15:01,117-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [226d75] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 32979254, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:15:01,127-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [4cc00193] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:15:01,128-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4cc00193] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4d7b53ad] Autorecovering 2 hosts
>2018-05-29 08:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4d7b53ad] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:20:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [72dec653] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:20:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [72dec653] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:20:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [72dec653] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:20:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [72dec653] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:20:00,052-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [72dec653] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5fdf15b5
>2018-05-29 08:20:00,055-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [72dec653] FINISH, SetVdsStatusVDSCommand, log id: 5fdf15b5
>2018-05-29 08:20:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [72dec653] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:20:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [72dec653] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:20:00,058-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [72dec653] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:20:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6db25422] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:20:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6db25422] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:20:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6db25422] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:20:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6db25422] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:20:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6db25422] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5295a716
>2018-05-29 08:20:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6db25422] FINISH, SetVdsStatusVDSCommand, log id: 5295a716
>2018-05-29 08:20:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [6db25422] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6874a49b
>2018-05-29 08:20:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [6db25422] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6874a49b
>2018-05-29 08:20:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6db25422] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:20:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6db25422] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:20:01,670-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4027c379] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 759fa88c
>2018-05-29 08:20:01,672-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4027c379] FINISH, GetHardwareInfoVDSCommand, log id: 759fa88c
>2018-05-29 08:20:01,685-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3075636d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:20:01,693-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [7f9891dc] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:20:01,698-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [49f07ee9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:20:01,704-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [49f07ee9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 715b5d89
>2018-05-29 08:20:01,756-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [31354382] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 29200aa8
>2018-05-29 08:20:01,758-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [31354382] FINISH, GetHardwareInfoVDSCommand, log id: 29200aa8
>2018-05-29 08:20:01,769-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4cf1fbf8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:20:01,774-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [103c8543] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:20:01,778-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [75b6cd61] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:20:01,784-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [75b6cd61] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3c2e30
>2018-05-29 08:20:02,080-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49f07ee9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:20:02,080-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [49f07ee9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 715b5d89
>2018-05-29 08:20:02,084-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49f07ee9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:20:02,085-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [49f07ee9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:20:02,089-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49f07ee9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:20:02,089-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [49f07ee9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:20:02,093-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49f07ee9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:20:02,094-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [49f07ee9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:20:02,096-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [49f07ee9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:20:02,099-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49f07ee9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 49f07ee9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:20:02,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5abbc84b
>2018-05-29 08:20:02,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:20:02,152-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75b6cd61] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:20:02,152-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [75b6cd61] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3c2e30
>2018-05-29 08:20:02,156-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75b6cd61] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:20:02,157-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [75b6cd61] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:20:02,160-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75b6cd61] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:20:02,161-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [75b6cd61] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:20:02,165-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75b6cd61] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:20:02,165-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [75b6cd61] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:20:02,168-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [75b6cd61] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:20:02,170-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75b6cd61] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 75b6cd61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:20:02,203-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:20:02,203-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:20:02,203-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:20:02,203-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:20:02,203-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5abbc84b
>2018-05-29 08:20:02,203-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:20:02,205-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 31e8df0d
>2018-05-29 08:20:02,205-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:20:02,216-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [49da411] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:20:02,217-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [49da411] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3e8faefb
>2018-05-29 08:20:02,219-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [49da411] FINISH, SetVdsStatusVDSCommand, log id: 3e8faefb
>2018-05-29 08:20:02,250-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49da411] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 49da411, Job ID: 98df4ce9-9100-42e9-bd3d-56c193a4c18e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:20:02,255-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49da411] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7f9891dc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:20:02,257-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49da411] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7f9891dc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:20:02,267-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [22cab21f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:20:02,268-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [22cab21f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:20:02,302-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:20:02,302-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:20:02,302-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:20:02,302-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:20:02,302-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 31e8df0d
>2018-05-29 08:20:02,302-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:20:02,314-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [658a78a4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:20:02,315-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [658a78a4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 24b11df3
>2018-05-29 08:20:02,317-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [658a78a4] FINISH, SetVdsStatusVDSCommand, log id: 24b11df3
>2018-05-29 08:20:02,323-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [658a78a4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 658a78a4, Job ID: c384ac7e-920c-4401-ac96-0210e15757ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:20:02,327-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [658a78a4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 103c8543, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:20:02,329-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [658a78a4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 103c8543, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:20:02,340-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [16e99fc7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:20:02,341-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [16e99fc7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [984de79] Autorecovering 2 hosts
>2018-05-29 08:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [984de79] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:25:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [139ca2ec] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [139ca2ec] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [139ca2ec] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [139ca2ec] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:25:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [139ca2ec] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42b2437e
>2018-05-29 08:25:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [139ca2ec] FINISH, SetVdsStatusVDSCommand, log id: 42b2437e
>2018-05-29 08:25:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [139ca2ec] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:25:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [139ca2ec] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:25:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [139ca2ec] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:25:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7389598f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7389598f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7389598f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7389598f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:25:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7389598f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18f541f
>2018-05-29 08:25:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7389598f] FINISH, SetVdsStatusVDSCommand, log id: 18f541f
>2018-05-29 08:25:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7389598f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6de603ed
>2018-05-29 08:25:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7389598f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6de603ed
>2018-05-29 08:25:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7389598f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:25:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7389598f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:25:03,704-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [ee613cf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6a56bf
>2018-05-29 08:25:03,705-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [ee613cf] FINISH, GetHardwareInfoVDSCommand, log id: 6a56bf
>2018-05-29 08:25:03,719-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [6f136aa1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:25:03,727-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [4a019873] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:25:03,732-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [1545f8e0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:25:03,743-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [1545f8e0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 159c5d45
>2018-05-29 08:25:03,786-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [41f7ac0e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 419a54b3
>2018-05-29 08:25:03,788-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [41f7ac0e] FINISH, GetHardwareInfoVDSCommand, log id: 419a54b3
>2018-05-29 08:25:03,800-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [36f33edf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:25:03,804-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [4ace637] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:25:03,809-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [9c28303] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:25:03,815-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [9c28303] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 78ca2221
>2018-05-29 08:25:04,124-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1545f8e0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:25:04,124-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [1545f8e0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 159c5d45
>2018-05-29 08:25:04,127-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1545f8e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:25:04,128-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1545f8e0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:25:04,132-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1545f8e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:25:04,133-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1545f8e0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:25:04,137-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1545f8e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:25:04,138-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1545f8e0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:25:04,140-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [1545f8e0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:25:04,143-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1545f8e0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1545f8e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:25:04,169-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 53c9da08
>2018-05-29 08:25:04,169-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:25:04,188-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9c28303] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:25:04,188-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [9c28303] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 78ca2221
>2018-05-29 08:25:04,192-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9c28303] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:25:04,193-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [9c28303] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:25:04,196-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9c28303] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:25:04,197-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [9c28303] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:25:04,201-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9c28303] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:25:04,202-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [9c28303] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:25:04,205-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [9c28303] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:25:04,207-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9c28303] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 9c28303, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:25:04,265-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:25:04,265-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:25:04,265-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:25:04,265-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:25:04,265-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 53c9da08
>2018-05-29 08:25:04,265-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:25:04,267-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5412ed7a
>2018-05-29 08:25:04,267-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:25:04,279-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [3ecc6359] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:25:04,280-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3ecc6359] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ca50e2f
>2018-05-29 08:25:04,282-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3ecc6359] FINISH, SetVdsStatusVDSCommand, log id: 2ca50e2f
>2018-05-29 08:25:04,314-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3ecc6359] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3ecc6359, Job ID: 7d739f02-0dfd-4c6d-b1a8-32d7769afe02, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:25:04,318-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3ecc6359] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4a019873, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:25:04,320-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3ecc6359] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4a019873, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:25:04,331-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [782dee07] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:25:04,332-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [782dee07] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:25:04,370-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:25:04,370-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:25:04,370-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:25:04,370-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:25:04,370-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5412ed7a
>2018-05-29 08:25:04,370-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:25:04,383-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [1f0b6a79] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:25:04,385-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1f0b6a79] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ded7bb4
>2018-05-29 08:25:04,387-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1f0b6a79] FINISH, SetVdsStatusVDSCommand, log id: 2ded7bb4
>2018-05-29 08:25:04,392-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1f0b6a79] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1f0b6a79, Job ID: 9307b87a-00de-4ed4-9f5a-f46beffb0da7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:25:04,397-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1f0b6a79] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4ace637, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:25:04,399-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1f0b6a79] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4ace637, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:25:04,409-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [7358c820] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:25:04,410-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [7358c820] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [617f4d1d] Autorecovering 2 hosts
>2018-05-29 08:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [617f4d1d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:30:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [258f6b9b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:30:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [258f6b9b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:30:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [258f6b9b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:30:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [258f6b9b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:30:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [258f6b9b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72c26b77
>2018-05-29 08:30:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [258f6b9b] FINISH, SetVdsStatusVDSCommand, log id: 72c26b77
>2018-05-29 08:30:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [258f6b9b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:30:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [258f6b9b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:30:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [258f6b9b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:30:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3ac085b5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:30:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3ac085b5] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:30:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3ac085b5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:30:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3ac085b5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:30:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3ac085b5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f565838
>2018-05-29 08:30:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3ac085b5] FINISH, SetVdsStatusVDSCommand, log id: 3f565838
>2018-05-29 08:30:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [3ac085b5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 56c63ca0
>2018-05-29 08:30:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [3ac085b5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 56c63ca0
>2018-05-29 08:30:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3ac085b5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:30:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3ac085b5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:30:02,004-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [12bc242c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7613b19e
>2018-05-29 08:30:02,005-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [12bc242c] FINISH, GetHardwareInfoVDSCommand, log id: 7613b19e
>2018-05-29 08:30:02,019-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [5ba8d006] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:30:02,027-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [69a58ca1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:30:02,032-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [4802da5c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:30:02,044-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [4802da5c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: df9ebbc
>2018-05-29 08:30:02,055-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3c2b7921] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 303c8da
>2018-05-29 08:30:02,057-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3c2b7921] FINISH, GetHardwareInfoVDSCommand, log id: 303c8da
>2018-05-29 08:30:02,069-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6f35ce96] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:30:02,073-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [2e616128] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:30:02,078-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [32d9495c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:30:02,084-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [32d9495c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22e4e754
>2018-05-29 08:30:02,426-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4802da5c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:30:02,426-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [4802da5c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: df9ebbc
>2018-05-29 08:30:02,430-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4802da5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:30:02,431-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [4802da5c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:30:02,434-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4802da5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:30:02,435-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [4802da5c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:30:02,439-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4802da5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:30:02,439-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [4802da5c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:30:02,442-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [4802da5c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:30:02,445-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4802da5c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4802da5c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:30:02,451-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 496bd3d
>2018-05-29 08:30:02,451-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:30:02,550-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:30:02,550-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:30:02,550-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:30:02,550-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:30:02,550-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: 496bd3d
>2018-05-29 08:30:02,550-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:30:02,563-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [26c2ce21] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:30:02,564-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [26c2ce21] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 35e8d350
>2018-05-29 08:30:02,566-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [26c2ce21] FINISH, SetVdsStatusVDSCommand, log id: 35e8d350
>2018-05-29 08:30:02,597-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26c2ce21] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 26c2ce21, Job ID: d0f2eaeb-1023-4965-a191-920aa961e3bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:30:02,602-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26c2ce21] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 69a58ca1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:30:02,605-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26c2ce21] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 69a58ca1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:30:02,615-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [12432a4a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:30:02,616-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [12432a4a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:30:03,452-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [32d9495c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:30:03,452-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [32d9495c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 22e4e754
>2018-05-29 08:30:03,456-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [32d9495c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:30:03,457-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [32d9495c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:30:03,461-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [32d9495c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:30:03,461-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [32d9495c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:30:03,465-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [32d9495c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:30:03,466-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [32d9495c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:30:03,469-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [32d9495c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:30:03,471-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [32d9495c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 32d9495c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:30:03,474-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 57fb8dd6
>2018-05-29 08:30:03,474-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:30:03,571-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:30:03,571-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:30:03,571-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:30:03,571-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:30:03,571-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 57fb8dd6
>2018-05-29 08:30:03,571-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:30:03,583-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [38afc133] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:30:03,584-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [38afc133] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1181ea1a
>2018-05-29 08:30:03,587-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [38afc133] FINISH, SetVdsStatusVDSCommand, log id: 1181ea1a
>2018-05-29 08:30:03,616-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [38afc133] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 38afc133, Job ID: b8655e6b-6879-427c-a0f2-094fa1dac7ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:30:03,621-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [38afc133] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2e616128, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:30:03,623-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [38afc133] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2e616128, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:30:03,634-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [66db3f88] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:30:03,635-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [66db3f88] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3995d9a1] Autorecovering 2 hosts
>2018-05-29 08:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3995d9a1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:35:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [383d4d67] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:35:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [383d4d67] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:35:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [383d4d67] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:35:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [383d4d67] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:35:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [383d4d67] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a8a81f2
>2018-05-29 08:35:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [383d4d67] FINISH, SetVdsStatusVDSCommand, log id: 5a8a81f2
>2018-05-29 08:35:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [383d4d67] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:35:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [383d4d67] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:35:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [383d4d67] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:35:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [d8818d1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:35:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [d8818d1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:35:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [d8818d1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:35:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [d8818d1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:35:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [d8818d1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 572f5307
>2018-05-29 08:35:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [d8818d1] FINISH, SetVdsStatusVDSCommand, log id: 572f5307
>2018-05-29 08:35:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [d8818d1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6c4509f8
>2018-05-29 08:35:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [d8818d1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6c4509f8
>2018-05-29 08:35:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [d8818d1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:35:00,102-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [d8818d1] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: d8818d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-29 08:35:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [d8818d1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:35:00,215-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [58d6eb99] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 11e619ef
>2018-05-29 08:35:00,217-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [58d6eb99] FINISH, GetHardwareInfoVDSCommand, log id: 11e619ef
>2018-05-29 08:35:00,229-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [736bfff3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:35:00,238-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [3b658777] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:35:00,243-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [75e9226e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:35:00,301-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [75e9226e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 90bb794
>2018-05-29 08:35:00,683-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75e9226e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:35:00,683-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [75e9226e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 90bb794
>2018-05-29 08:35:00,687-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75e9226e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:35:00,688-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [75e9226e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:35:00,692-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75e9226e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:35:00,693-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [75e9226e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:35:00,696-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75e9226e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:35:00,697-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [75e9226e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:35:00,700-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [75e9226e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:35:00,702-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75e9226e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 75e9226e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:35:00,709-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: c43c021
>2018-05-29 08:35:00,709-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:35:00,807-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:35:00,807-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:35:00,807-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:35:00,807-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:35:00,808-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: c43c021
>2018-05-29 08:35:00,808-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:35:00,821-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [27fac8b1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:35:00,822-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [27fac8b1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7126a167
>2018-05-29 08:35:00,824-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [27fac8b1] FINISH, SetVdsStatusVDSCommand, log id: 7126a167
>2018-05-29 08:35:00,856-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [27fac8b1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 27fac8b1, Job ID: be380688-4df4-42b8-b9b3-3c3ac2d79315, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:35:00,861-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [27fac8b1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b658777, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:35:00,863-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [27fac8b1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b658777, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:35:00,874-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [5ff917] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:35:00,875-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [5ff917] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:35:01,286-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7eb59dce] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 455ee8a1
>2018-05-29 08:35:01,287-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7eb59dce] FINISH, GetHardwareInfoVDSCommand, log id: 455ee8a1
>2018-05-29 08:35:01,301-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [1658bae0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:35:01,305-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [10d3a862] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:35:01,310-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [424fb27e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:35:01,317-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [424fb27e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 11a624bf
>2018-05-29 08:35:01,691-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [424fb27e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:35:01,691-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [424fb27e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 11a624bf
>2018-05-29 08:35:01,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [424fb27e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:35:01,696-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [424fb27e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:35:01,699-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [424fb27e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:35:01,700-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [424fb27e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:35:01,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [424fb27e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:35:01,705-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [424fb27e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:35:01,707-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [424fb27e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:35:01,710-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [424fb27e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 424fb27e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:35:01,713-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7533dfe2
>2018-05-29 08:35:01,713-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:35:01,812-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:35:01,812-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:35:01,812-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:35:01,812-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:35:01,812-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7533dfe2
>2018-05-29 08:35:01,812-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:35:01,825-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [1ec5209d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:35:01,827-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1ec5209d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 386fa481
>2018-05-29 08:35:01,829-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1ec5209d] FINISH, SetVdsStatusVDSCommand, log id: 386fa481
>2018-05-29 08:35:01,835-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ec5209d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1ec5209d, Job ID: dd818572-2f60-4aea-8d6b-39dc165af773, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:35:01,839-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ec5209d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 10d3a862, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:35:01,841-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ec5209d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 10d3a862, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:35:01,851-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [1472accc] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:35:01,853-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [1472accc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:40:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [43027f99] Autorecovering 2 hosts
>2018-05-29 08:40:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [43027f99] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:40:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7a699e85] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:40:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7a699e85] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:40:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7a699e85] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7a699e85] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:40:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7a699e85] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 556cc032
>2018-05-29 08:40:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7a699e85] FINISH, SetVdsStatusVDSCommand, log id: 556cc032
>2018-05-29 08:40:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7a699e85] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:40:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7a699e85] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:40:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7a699e85] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:40:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7e970117] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:40:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7e970117] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:40:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7e970117] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:40:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7e970117] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:40:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7e970117] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 156aecff
>2018-05-29 08:40:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7e970117] FINISH, SetVdsStatusVDSCommand, log id: 156aecff
>2018-05-29 08:40:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [7e970117] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 334a4d62
>2018-05-29 08:40:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [7e970117] FINISH, SetHaMaintenanceModeVDSCommand, log id: 334a4d62
>2018-05-29 08:40:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7e970117] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:40:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7e970117] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:40:01,594-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [475b6bda] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 66f5bd9e
>2018-05-29 08:40:01,596-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [475b6bda] FINISH, GetHardwareInfoVDSCommand, log id: 66f5bd9e
>2018-05-29 08:40:01,612-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [55fd7d15] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:40:01,620-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [688d2129] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:40:01,625-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7dabb868] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:40:01,638-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7dabb868] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 164b182
>2018-05-29 08:40:02,022-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7dabb868] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:40:02,022-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7dabb868] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 164b182
>2018-05-29 08:40:02,026-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7dabb868] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:40:02,026-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7dabb868] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:40:02,030-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7dabb868] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:40:02,031-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7dabb868] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:40:02,035-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7dabb868] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:40:02,035-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7dabb868] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:40:02,038-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7dabb868] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:40:02,041-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7dabb868] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7dabb868, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:40:02,048-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 277f7849
>2018-05-29 08:40:02,048-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:40:02,146-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:40:02,146-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:40:02,146-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:40:02,146-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:40:02,146-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 277f7849
>2018-05-29 08:40:02,146-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:40:02,161-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [1776bf79] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:40:02,162-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1776bf79] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 363ff07e
>2018-05-29 08:40:02,164-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1776bf79] FINISH, SetVdsStatusVDSCommand, log id: 363ff07e
>2018-05-29 08:40:02,196-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1776bf79] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1776bf79, Job ID: 1fb5acb2-c4a5-4fcb-a2ba-ce415f6793df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:40:02,200-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1776bf79] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 688d2129, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:40:02,203-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1776bf79] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 688d2129, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:40:02,214-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [42545303] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:40:02,215-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [42545303] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:40:02,529-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6ef8caad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6b7c49e2
>2018-05-29 08:40:02,531-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6ef8caad] FINISH, GetHardwareInfoVDSCommand, log id: 6b7c49e2
>2018-05-29 08:40:02,544-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6a145be6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:40:02,548-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3d588d26] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:40:02,553-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [13378390] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:40:02,561-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [13378390] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6534c065
>2018-05-29 08:40:03,929-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [13378390] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:40:03,929-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [13378390] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6534c065
>2018-05-29 08:40:03,933-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [13378390] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:40:03,933-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [13378390] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:40:03,937-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [13378390] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:40:03,938-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [13378390] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:40:03,941-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [13378390] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:40:03,942-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [13378390] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:40:03,945-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [13378390] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:40:03,947-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [13378390] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 13378390, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:40:03,951-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 79d4d4b6
>2018-05-29 08:40:03,951-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:40:04,048-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:40:04,048-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:40:04,048-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:40:04,048-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:40:04,048-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 79d4d4b6
>2018-05-29 08:40:04,048-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:40:04,061-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [6ebca3bc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:40:04,062-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6ebca3bc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7194066d
>2018-05-29 08:40:04,064-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6ebca3bc] FINISH, SetVdsStatusVDSCommand, log id: 7194066d
>2018-05-29 08:40:04,070-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ebca3bc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6ebca3bc, Job ID: 7fa1fdf6-09fb-4cd5-9ef3-1b11677df328, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:40:04,074-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ebca3bc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3d588d26, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:40:04,076-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ebca3bc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3d588d26, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:40:04,087-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [6c0f2fa] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:40:04,088-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [6c0f2fa] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [4f3e3e95] Autorecovering 2 hosts
>2018-05-29 08:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [4f3e3e95] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:45:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70933836] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:45:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70933836] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:45:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70933836] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:45:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70933836] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:45:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [70933836] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ae2aa55
>2018-05-29 08:45:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [70933836] FINISH, SetVdsStatusVDSCommand, log id: 6ae2aa55
>2018-05-29 08:45:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70933836] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:45:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70933836] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:45:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [70933836] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:45:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [334bb122] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:45:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [334bb122] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:45:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [334bb122] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:45:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [334bb122] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:45:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [334bb122] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 40baed79
>2018-05-29 08:45:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [334bb122] FINISH, SetVdsStatusVDSCommand, log id: 40baed79
>2018-05-29 08:45:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [334bb122] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 660f0335
>2018-05-29 08:45:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [334bb122] FINISH, SetHaMaintenanceModeVDSCommand, log id: 660f0335
>2018-05-29 08:45:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [334bb122] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:45:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [334bb122] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:45:01,808-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [e0c6b78] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 20e223f8
>2018-05-29 08:45:01,810-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [e0c6b78] FINISH, GetHardwareInfoVDSCommand, log id: 20e223f8
>2018-05-29 08:45:01,823-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [1eecd5e7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:45:01,832-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [12dfbeec] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:45:01,837-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [ab57446] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:45:01,849-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [ab57446] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 212bb142
>2018-05-29 08:45:02,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ab57446] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:45:02,224-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [ab57446] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 212bb142
>2018-05-29 08:45:02,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ab57446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:45:02,229-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [ab57446] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:45:02,233-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ab57446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:45:02,234-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [ab57446] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:45:02,238-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ab57446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:45:02,238-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [ab57446] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:45:02,241-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [ab57446] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:45:02,244-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ab57446] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: ab57446, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:45:02,251-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4fec5132
>2018-05-29 08:45:02,251-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:45:02,350-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:45:02,350-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:45:02,350-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:45:02,350-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:45:02,350-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4fec5132
>2018-05-29 08:45:02,350-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:45:02,364-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [1db22768] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:45:02,365-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1db22768] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3cbcd21e
>2018-05-29 08:45:02,368-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1db22768] FINISH, SetVdsStatusVDSCommand, log id: 3cbcd21e
>2018-05-29 08:45:02,399-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1db22768] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1db22768, Job ID: 77935ae6-043b-4a0b-bd16-e2991106d975, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:45:02,404-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1db22768] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 12dfbeec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:45:02,406-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1db22768] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 12dfbeec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:45:02,417-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [1affdb96] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:45:02,418-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [1affdb96] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:45:03,010-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7659437e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 24170f54
>2018-05-29 08:45:03,012-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7659437e] FINISH, GetHardwareInfoVDSCommand, log id: 24170f54
>2018-05-29 08:45:03,024-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [1a041b4f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:45:03,029-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [5a228065] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:45:03,034-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3362a530] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:45:03,041-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3362a530] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 554a4b5b
>2018-05-29 08:45:03,415-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3362a530] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:45:03,415-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3362a530] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 554a4b5b
>2018-05-29 08:45:03,420-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3362a530] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:45:03,420-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3362a530] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:45:03,424-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3362a530] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:45:03,425-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3362a530] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:45:03,429-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3362a530] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:45:03,430-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3362a530] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:45:03,433-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3362a530] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:45:03,435-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3362a530] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3362a530, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:45:03,438-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 728914df
>2018-05-29 08:45:03,438-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:45:03,535-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:45:03,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:45:03,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:45:03,535-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:45:03,536-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 728914df
>2018-05-29 08:45:03,536-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:45:03,549-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [e544d16] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:45:03,550-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [e544d16] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50572791
>2018-05-29 08:45:03,552-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [e544d16] FINISH, SetVdsStatusVDSCommand, log id: 50572791
>2018-05-29 08:45:03,558-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [e544d16] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: e544d16, Job ID: b6071715-4686-496e-be7c-308ad4a834d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:45:03,562-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [e544d16] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5a228065, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:45:03,565-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [e544d16] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5a228065, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:45:03,575-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [267ba407] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:45:03,576-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [267ba407] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:50:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2b0a949e] Autorecovering 2 hosts
>2018-05-29 08:50:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2b0a949e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:50:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1aafcefe] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:50:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1aafcefe] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:50:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1aafcefe] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:50:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1aafcefe] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:50:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1aafcefe] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3490ef02
>2018-05-29 08:50:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1aafcefe] FINISH, SetVdsStatusVDSCommand, log id: 3490ef02
>2018-05-29 08:50:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1aafcefe] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:50:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1aafcefe] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:50:00,073-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [1aafcefe] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:50:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3b8c8dde] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:50:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3b8c8dde] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:50:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3b8c8dde] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:50:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3b8c8dde] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:50:00,086-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3b8c8dde] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3af55209
>2018-05-29 08:50:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3b8c8dde] FINISH, SetVdsStatusVDSCommand, log id: 3af55209
>2018-05-29 08:50:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [3b8c8dde] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3638e1b1
>2018-05-29 08:50:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [3b8c8dde] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3638e1b1
>2018-05-29 08:50:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3b8c8dde] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:50:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3b8c8dde] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:50:01,201-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [b155955] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4d73f372
>2018-05-29 08:50:01,202-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [b155955] FINISH, GetHardwareInfoVDSCommand, log id: 4d73f372
>2018-05-29 08:50:01,215-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [78cbd282] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:50:01,220-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [163cabd8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:50:01,225-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3593c75b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:50:01,238-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3593c75b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c1365a3
>2018-05-29 08:50:01,623-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3593c75b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:50:01,623-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3593c75b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c1365a3
>2018-05-29 08:50:01,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3593c75b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:50:01,627-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3593c75b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:50:01,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3593c75b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:50:01,632-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3593c75b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:50:01,636-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3593c75b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:50:01,636-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3593c75b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:50:01,639-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3593c75b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:50:01,642-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3593c75b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3593c75b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:50:01,648-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5d3c131d
>2018-05-29 08:50:01,648-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:50:01,748-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:50:01,748-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:50:01,748-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:50:01,748-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:50:01,748-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5d3c131d
>2018-05-29 08:50:01,748-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:50:01,761-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [23e7c5f7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:50:01,763-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [23e7c5f7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5aa9d24e
>2018-05-29 08:50:01,765-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [23e7c5f7] FINISH, SetVdsStatusVDSCommand, log id: 5aa9d24e
>2018-05-29 08:50:01,796-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [23e7c5f7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 23e7c5f7, Job ID: fac3304a-f2e0-4d5e-854b-d9737e0df58e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:50:01,801-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [23e7c5f7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 163cabd8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:50:01,803-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [23e7c5f7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 163cabd8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:50:01,813-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [6064e00f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:50:01,815-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [6064e00f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:50:03,164-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [2cd282d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5474c9ca
>2018-05-29 08:50:03,166-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [2cd282d] FINISH, GetHardwareInfoVDSCommand, log id: 5474c9ca
>2018-05-29 08:50:03,180-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [1801f6f9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:50:03,184-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [ed20748] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:50:03,189-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [93413b9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:50:03,196-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [93413b9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 78bde303
>2018-05-29 08:50:04,563-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [93413b9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:50:04,563-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [93413b9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 78bde303
>2018-05-29 08:50:04,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [93413b9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:50:04,568-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [93413b9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:50:04,572-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [93413b9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:50:04,572-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [93413b9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:50:04,576-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [93413b9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:50:04,577-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [93413b9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:50:04,580-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [93413b9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:50:04,583-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [93413b9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 93413b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:50:04,589-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 785b793a
>2018-05-29 08:50:04,589-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:50:04,685-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:50:04,685-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:50:04,685-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:50:04,685-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:50:04,685-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 785b793a
>2018-05-29 08:50:04,685-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:50:04,698-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [358e5112] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:50:04,700-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [358e5112] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 349a8bd4
>2018-05-29 08:50:04,702-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [358e5112] FINISH, SetVdsStatusVDSCommand, log id: 349a8bd4
>2018-05-29 08:50:04,710-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [358e5112] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 358e5112, Job ID: 740b8bc2-144a-4ad7-b40b-5060ebe1dc40, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:50:04,714-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [358e5112] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: ed20748, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:50:04,716-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [358e5112] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: ed20748, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:50:04,727-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [553cdb0e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:50:04,728-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [553cdb0e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:54:42,967-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler6) [2480d1a] Attempting to update VMs/Templates Ovf.
>2018-05-29 08:54:42,969-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 08:54:42,969-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 08:54:42,970-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:54:42,974-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 08:54:42,977-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 08:54:42,977-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 08:54:42,978-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 08:54:42,978-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 08:54:42,978-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 08:54:42,979-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 08:55:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2587f861] Autorecovering 2 hosts
>2018-05-29 08:55:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2587f861] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:55:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [755ade9c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:55:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [755ade9c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:55:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [755ade9c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:55:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [755ade9c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:55:00,048-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [755ade9c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5b750b0c
>2018-05-29 08:55:00,051-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [755ade9c] FINISH, SetVdsStatusVDSCommand, log id: 5b750b0c
>2018-05-29 08:55:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [755ade9c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:55:00,056-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [755ade9c] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 755ade9c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-29 08:55:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [755ade9c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 08:55:00,057-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [755ade9c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:55:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [38f4e2ea] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [38f4e2ea] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 08:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [38f4e2ea] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [38f4e2ea] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:55:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [38f4e2ea] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6249396
>2018-05-29 08:55:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [38f4e2ea] FINISH, SetVdsStatusVDSCommand, log id: 6249396
>2018-05-29 08:55:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [38f4e2ea] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4431c558
>2018-05-29 08:55:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [38f4e2ea] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4431c558
>2018-05-29 08:55:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [38f4e2ea] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 08:55:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [38f4e2ea] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 08:55:02,378-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [416dccd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 276d35f2
>2018-05-29 08:55:02,380-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [416dccd] FINISH, GetHardwareInfoVDSCommand, log id: 276d35f2
>2018-05-29 08:55:02,393-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [61c25662] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:55:02,399-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [69645bbe] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1b8c623
>2018-05-29 08:55:02,401-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [69645bbe] FINISH, GetHardwareInfoVDSCommand, log id: 1b8c623
>2018-05-29 08:55:02,402-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [70570919] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:55:02,408-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [45db4d1b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:55:02,413-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [b2c5cf5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:55:02,416-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [45db4d1b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3e4cdb5a
>2018-05-29 08:55:02,418-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5910726] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:55:02,423-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [213eb676] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 08:55:02,428-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [213eb676] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5b267189
>2018-05-29 08:55:02,785-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [45db4d1b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:55:02,785-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [45db4d1b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3e4cdb5a
>2018-05-29 08:55:02,789-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [45db4d1b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:55:02,789-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [45db4d1b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:55:02,793-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [45db4d1b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:55:02,794-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [45db4d1b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:55:02,798-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [45db4d1b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:55:02,798-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [45db4d1b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:55:02,802-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [45db4d1b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:55:02,805-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [45db4d1b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 45db4d1b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:55:02,811-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [213eb676] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 08:55:02,811-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [213eb676] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5b267189
>2018-05-29 08:55:02,812-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2f2f1a17
>2018-05-29 08:55:02,812-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:55:02,815-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [213eb676] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:55:02,816-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [213eb676] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:55:02,820-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [213eb676] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:55:02,820-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [213eb676] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:55:02,824-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [213eb676] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 08:55:02,825-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [213eb676] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 08:55:02,827-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [213eb676] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 08:55:02,830-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [213eb676] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 213eb676, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 08:55:02,908-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:55:02,908-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:55:02,908-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:55:02,908-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:55:02,908-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2f2f1a17
>2018-05-29 08:55:02,908-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:55:02,909-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 38113a51
>2018-05-29 08:55:02,910-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 08:55:02,921-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [7271dfdd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:55:02,922-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7271dfdd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 570c319e
>2018-05-29 08:55:02,924-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7271dfdd] FINISH, SetVdsStatusVDSCommand, log id: 570c319e
>2018-05-29 08:55:02,954-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7271dfdd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7271dfdd, Job ID: d4cda4bd-bf8d-4173-9c28-9d903cc302f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:55:02,958-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7271dfdd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 70570919, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:55:02,961-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7271dfdd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 70570919, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:55:02,971-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [208a8419] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 08:55:02,972-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [208a8419] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 08:55:03,010-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:55:03,010-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 08:55:03,010-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 08:55:03,010-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 08:55:03,010-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 38113a51
>2018-05-29 08:55:03,010-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 08:55:03,022-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [42feb506] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:55:03,023-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [42feb506] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 652373c9
>2018-05-29 08:55:03,026-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [42feb506] FINISH, SetVdsStatusVDSCommand, log id: 652373c9
>2018-05-29 08:55:03,031-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42feb506] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 42feb506, Job ID: 66bd4e3a-b5f5-4e6c-a142-525491bbed9b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 08:55:03,036-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42feb506] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5910726, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 08:55:03,038-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42feb506] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5910726, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 08:55:03,048-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [51fdb5de] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 08:55:03,050-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [51fdb5de] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:00:00,006-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [28399481] Autorecovering 2 hosts
>2018-05-29 09:00:00,006-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [28399481] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:00:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3491c1ef] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:00:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3491c1ef] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:00:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3491c1ef] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:00:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3491c1ef] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:00:00,049-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3491c1ef] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 310561f5
>2018-05-29 09:00:00,052-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3491c1ef] FINISH, SetVdsStatusVDSCommand, log id: 310561f5
>2018-05-29 09:00:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3491c1ef] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:00:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3491c1ef] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:00:00,055-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [3491c1ef] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:00:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [890723a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:00:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [890723a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:00:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [890723a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:00:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [890723a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:00:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [890723a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 244e932f
>2018-05-29 09:00:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [890723a] FINISH, SetVdsStatusVDSCommand, log id: 244e932f
>2018-05-29 09:00:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [890723a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5af2df07
>2018-05-29 09:00:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [890723a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5af2df07
>2018-05-29 09:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [890723a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:00:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [890723a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:00:00,608-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [74e54887] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 48ca657a
>2018-05-29 09:00:00,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [74e54887] FINISH, GetHardwareInfoVDSCommand, log id: 48ca657a
>2018-05-29 09:00:00,624-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [626ee555] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:00:00,632-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [4a9dd592] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:00:00,637-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4d08f0b1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:00:00,696-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4d08f0b1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 201577c3
>2018-05-29 09:00:00,709-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4201690b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 45189e37
>2018-05-29 09:00:00,711-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4201690b] FINISH, GetHardwareInfoVDSCommand, log id: 45189e37
>2018-05-29 09:00:00,725-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [2544550d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:00:00,730-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [d38ddd8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:00:00,735-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [344b7562] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:00:00,741-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [344b7562] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7ef65963
>2018-05-29 09:00:01,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d08f0b1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:00:01,081-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4d08f0b1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 201577c3
>2018-05-29 09:00:01,085-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d08f0b1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:00:01,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d08f0b1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:00:01,089-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d08f0b1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:00:01,090-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d08f0b1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:00:01,094-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d08f0b1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:00:01,095-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d08f0b1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:00:01,097-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4d08f0b1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:00:01,100-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d08f0b1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d08f0b1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:00:01,107-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [344b7562] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:00:01,107-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [344b7562] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7ef65963
>2018-05-29 09:00:01,108-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 68138bcb
>2018-05-29 09:00:01,108-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:00:01,111-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [344b7562] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:00:01,112-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [344b7562] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:00:01,116-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [344b7562] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:00:01,116-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [344b7562] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:00:01,120-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [344b7562] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:00:01,121-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [344b7562] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:00:01,124-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [344b7562] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:00:01,126-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [344b7562] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 344b7562, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:00:01,206-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:00:01,206-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:00:01,206-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:00:01,206-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:00:01,206-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 68138bcb
>2018-05-29 09:00:01,206-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:00:01,212-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4665467e
>2018-05-29 09:00:01,212-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:00:01,220-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [2e308d47] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:00:01,221-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2e308d47] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12cec1f
>2018-05-29 09:00:01,224-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2e308d47] FINISH, SetVdsStatusVDSCommand, log id: 12cec1f
>2018-05-29 09:00:01,257-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e308d47] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2e308d47, Job ID: 4d744a32-0b8f-431f-8bf3-35840cd5e8bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:00:01,262-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e308d47] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4a9dd592, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:00:01,264-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e308d47] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4a9dd592, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:00:01,275-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [25dd0677] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:00:01,276-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [25dd0677] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:00:01,311-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:00:01,311-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:00:01,311-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:00:01,311-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:00:01,311-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4665467e
>2018-05-29 09:00:01,311-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:00:01,325-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [4e7f29b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:00:01,326-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4e7f29b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f475cc9
>2018-05-29 09:00:01,328-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4e7f29b] FINISH, SetVdsStatusVDSCommand, log id: 7f475cc9
>2018-05-29 09:00:01,334-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e7f29b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4e7f29b, Job ID: b7d92374-e15c-4a26-a317-366a9308775a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:00:01,338-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e7f29b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: d38ddd8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:00:01,340-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e7f29b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: d38ddd8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:00:01,351-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [1d04c49d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:00:01,352-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [1d04c49d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1d7a46fe] Autorecovering 2 hosts
>2018-05-29 09:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1d7a46fe] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:05:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ab9ad2e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:05:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ab9ad2e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ab9ad2e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ab9ad2e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:05:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3ab9ad2e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 180c0172
>2018-05-29 09:05:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3ab9ad2e] FINISH, SetVdsStatusVDSCommand, log id: 180c0172
>2018-05-29 09:05:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ab9ad2e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:05:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ab9ad2e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:05:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3ab9ad2e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:05:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b1d344] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b1d344] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b1d344] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b1d344] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:05:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [54b1d344] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 89bb85c
>2018-05-29 09:05:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [54b1d344] FINISH, SetVdsStatusVDSCommand, log id: 89bb85c
>2018-05-29 09:05:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [54b1d344] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2c098a5
>2018-05-29 09:05:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [54b1d344] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2c098a5
>2018-05-29 09:05:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b1d344] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:05:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b1d344] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:05:01,890-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [7c93cb65] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 30ec9e2d
>2018-05-29 09:05:01,891-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [7c93cb65] FINISH, GetHardwareInfoVDSCommand, log id: 30ec9e2d
>2018-05-29 09:05:01,906-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [7960f0da] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:05:01,915-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [56cbc57d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:05:01,920-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6356e22] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:05:01,931-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6356e22] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: c3e41c3
>2018-05-29 09:05:01,981-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [451d712d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4b0b554a
>2018-05-29 09:05:01,983-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [451d712d] FINISH, GetHardwareInfoVDSCommand, log id: 4b0b554a
>2018-05-29 09:05:01,996-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [3e5b7e49] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:05:02,000-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [69862cb4] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:05:02,005-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6c969340] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:05:02,011-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6c969340] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 10c08071
>2018-05-29 09:05:02,318-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6356e22] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:05:02,319-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6356e22] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: c3e41c3
>2018-05-29 09:05:02,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6356e22] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:05:02,323-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6356e22] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:05:02,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6356e22] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:05:02,328-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6356e22] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:05:02,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6356e22] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:05:02,333-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6356e22] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:05:02,336-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6356e22] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:05:02,339-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6356e22] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6356e22, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:05:02,345-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5bf9a434
>2018-05-29 09:05:02,345-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:05:02,376-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6c969340] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:05:02,376-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6c969340] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 10c08071
>2018-05-29 09:05:02,380-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6c969340] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:05:02,381-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6c969340] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:05:02,384-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6c969340] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:05:02,385-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6c969340] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:05:02,389-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6c969340] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:05:02,390-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6c969340] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:05:02,392-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6c969340] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:05:02,395-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6c969340] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6c969340, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:05:02,447-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:05:02,447-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:05:02,447-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:05:02,447-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:05:02,447-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5bf9a434
>2018-05-29 09:05:02,447-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:05:02,449-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 490e8fdc
>2018-05-29 09:05:02,449-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:05:02,461-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [942bc2b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:05:02,462-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [942bc2b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8fe8cfb
>2018-05-29 09:05:02,464-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [942bc2b] FINISH, SetVdsStatusVDSCommand, log id: 8fe8cfb
>2018-05-29 09:05:02,497-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [942bc2b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 942bc2b, Job ID: fc59ec3d-5997-468e-a2dc-0acb7832aadb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:05:02,502-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [942bc2b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 56cbc57d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:05:02,504-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [942bc2b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 56cbc57d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:05:02,515-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [5b57c60e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:05:02,516-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [5b57c60e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:05:02,548-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:05:02,548-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:05:02,548-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:05:02,548-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:05:02,548-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 490e8fdc
>2018-05-29 09:05:02,548-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:05:02,562-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [22d56af5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:05:02,563-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [22d56af5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25a63ea1
>2018-05-29 09:05:02,566-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [22d56af5] FINISH, SetVdsStatusVDSCommand, log id: 25a63ea1
>2018-05-29 09:05:02,598-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [22d56af5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22d56af5, Job ID: 00601919-57bf-4a9c-b700-d141670821d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:05:02,604-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [22d56af5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 69862cb4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:05:02,608-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [22d56af5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 69862cb4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:05:02,619-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [6b3d6efb] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:05:02,620-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [6b3d6efb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [17a5d615] Autorecovering 2 hosts
>2018-05-29 09:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [17a5d615] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:10:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5de5e5e2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5de5e5e2] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5de5e5e2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5de5e5e2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:10:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5de5e5e2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 229d371f
>2018-05-29 09:10:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5de5e5e2] FINISH, SetVdsStatusVDSCommand, log id: 229d371f
>2018-05-29 09:10:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5de5e5e2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:10:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5de5e5e2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:10:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5de5e5e2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:10:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2ca9735e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2ca9735e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2ca9735e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2ca9735e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:10:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2ca9735e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 60fccb83
>2018-05-29 09:10:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2ca9735e] FINISH, SetVdsStatusVDSCommand, log id: 60fccb83
>2018-05-29 09:10:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [2ca9735e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5194777f
>2018-05-29 09:10:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [2ca9735e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5194777f
>2018-05-29 09:10:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2ca9735e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:10:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2ca9735e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:10:00,277-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [358f6c7b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 76af16be
>2018-05-29 09:10:00,279-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [358f6c7b] FINISH, GetHardwareInfoVDSCommand, log id: 76af16be
>2018-05-29 09:10:00,293-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [4490b2a1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:10:00,301-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [2bb581e1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:10:00,306-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2d2d2c5a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:10:00,318-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2d2d2c5a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 643d63b8
>2018-05-29 09:10:00,696-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d2d2c5a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:10:00,696-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2d2d2c5a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 643d63b8
>2018-05-29 09:10:00,700-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d2d2c5a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:10:00,701-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2d2d2c5a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:10:00,705-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d2d2c5a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:10:00,706-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2d2d2c5a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:10:00,710-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d2d2c5a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:10:00,710-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2d2d2c5a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:10:00,713-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2d2d2c5a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:10:00,716-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d2d2c5a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2d2d2c5a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:10:00,722-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3e98eded
>2018-05-29 09:10:00,722-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:10:00,820-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:10:00,820-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:10:00,820-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:10:00,821-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:10:00,821-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3e98eded
>2018-05-29 09:10:00,821-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:10:00,835-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [2defbb28] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:10:00,836-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2defbb28] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2fc84ca6
>2018-05-29 09:10:00,838-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2defbb28] FINISH, SetVdsStatusVDSCommand, log id: 2fc84ca6
>2018-05-29 09:10:00,870-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2defbb28] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2defbb28, Job ID: b23d790e-5a0d-4783-b602-14143c0c21a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:10:00,875-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2defbb28] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2bb581e1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:10:00,877-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2defbb28] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2bb581e1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:10:00,888-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [2693cfc0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:10:00,889-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [2693cfc0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:10:03,130-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [12d8217f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 31a31ef8
>2018-05-29 09:10:03,132-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [12d8217f] FINISH, GetHardwareInfoVDSCommand, log id: 31a31ef8
>2018-05-29 09:10:03,146-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [75883f97] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:10:03,150-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [42c69a24] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:10:03,156-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [3bea2208] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:10:03,163-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [3bea2208] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 501b2d38
>2018-05-29 09:10:03,541-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3bea2208] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:10:03,541-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [3bea2208] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 501b2d38
>2018-05-29 09:10:03,546-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3bea2208] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:10:03,546-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3bea2208] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:10:03,550-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3bea2208] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:10:03,551-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3bea2208] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:10:03,555-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3bea2208] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:10:03,556-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3bea2208] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:10:03,558-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [3bea2208] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:10:03,561-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3bea2208] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3bea2208, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:10:03,564-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 13814250
>2018-05-29 09:10:03,564-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:10:03,663-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:10:03,663-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:10:03,664-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:10:03,664-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:10:03,664-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 13814250
>2018-05-29 09:10:03,664-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:10:03,677-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [67b57bd0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:10:03,678-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [67b57bd0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c31b080
>2018-05-29 09:10:03,680-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [67b57bd0] FINISH, SetVdsStatusVDSCommand, log id: 7c31b080
>2018-05-29 09:10:03,686-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [67b57bd0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 67b57bd0, Job ID: 525f86b5-1d1c-4b19-b9f3-dfc244616085, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:10:03,691-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [67b57bd0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 42c69a24, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:10:03,693-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [67b57bd0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 42c69a24, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:10:03,703-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [45ece9bc] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:10:03,704-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [45ece9bc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:15:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1bbc1178] Autorecovering 2 hosts
>2018-05-29 09:15:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1bbc1178] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:15:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [57cd2c14] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [57cd2c14] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [57cd2c14] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [57cd2c14] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:15:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [57cd2c14] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bff1161
>2018-05-29 09:15:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [57cd2c14] FINISH, SetVdsStatusVDSCommand, log id: 3bff1161
>2018-05-29 09:15:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [57cd2c14] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:15:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [57cd2c14] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:15:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [57cd2c14] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:15:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [c545014] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:15:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [c545014] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:15:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [c545014] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:15:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [c545014] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:15:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [c545014] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f552e8
>2018-05-29 09:15:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [c545014] FINISH, SetVdsStatusVDSCommand, log id: 1f552e8
>2018-05-29 09:15:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [c545014] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 43336618
>2018-05-29 09:15:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [c545014] FINISH, SetHaMaintenanceModeVDSCommand, log id: 43336618
>2018-05-29 09:15:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [c545014] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:15:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [c545014] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:15:01,339-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2595cb7e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 58e5708f
>2018-05-29 09:15:01,341-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2595cb7e] FINISH, GetHardwareInfoVDSCommand, log id: 58e5708f
>2018-05-29 09:15:01,355-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [62af264] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:15:01,363-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [28fd1222] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:15:01,369-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [73274eb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:15:01,381-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [73274eb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6ea0adcb
>2018-05-29 09:15:01,578-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [186fb4aa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7940be1d
>2018-05-29 09:15:01,580-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [186fb4aa] FINISH, GetHardwareInfoVDSCommand, log id: 7940be1d
>2018-05-29 09:15:01,592-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [1b77705f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:15:01,597-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [346eb320] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:15:01,602-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [69a48446] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:15:01,609-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [69a48446] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 52916e0e
>2018-05-29 09:15:01,768-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [73274eb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:15:01,768-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [73274eb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6ea0adcb
>2018-05-29 09:15:01,772-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [73274eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:15:01,773-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [73274eb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:15:01,777-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [73274eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:15:01,777-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [73274eb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:15:01,781-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [73274eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:15:01,782-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [73274eb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:15:01,785-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [73274eb] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:15:01,787-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [73274eb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 73274eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:15:01,794-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 635c270a
>2018-05-29 09:15:01,794-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:15:01,892-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:15:01,892-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:15:01,892-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:15:01,892-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:15:01,892-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 635c270a
>2018-05-29 09:15:01,892-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:15:01,906-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [575d9383] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:15:01,907-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [575d9383] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6083d0c8
>2018-05-29 09:15:01,910-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [575d9383] FINISH, SetVdsStatusVDSCommand, log id: 6083d0c8
>2018-05-29 09:15:01,942-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [575d9383] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 575d9383, Job ID: 78282384-13f0-476e-a6c8-771f2f9f0a15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:15:01,946-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [575d9383] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 28fd1222, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:15:01,949-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [575d9383] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 28fd1222, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:15:01,959-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [33b422d2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:15:01,960-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [33b422d2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:15:01,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69a48446] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:15:01,983-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [69a48446] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 52916e0e
>2018-05-29 09:15:01,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69a48446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:15:01,988-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [69a48446] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:15:01,992-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69a48446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:15:01,992-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [69a48446] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:15:01,996-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69a48446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:15:01,997-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [69a48446] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:15:02,000-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [69a48446] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:15:02,003-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69a48446] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 69a48446, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:15:02,006-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 74a34a9f
>2018-05-29 09:15:02,006-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:15:02,101-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:15:02,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:15:02,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:15:02,101-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:15:02,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 74a34a9f
>2018-05-29 09:15:02,101-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:15:02,114-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [3763799] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:15:02,115-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3763799] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2774127d
>2018-05-29 09:15:02,117-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3763799] FINISH, SetVdsStatusVDSCommand, log id: 2774127d
>2018-05-29 09:15:02,123-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3763799] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3763799, Job ID: d31a2453-6895-431c-a443-0a6222763b5c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:15:02,127-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3763799] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 346eb320, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:15:02,129-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3763799] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 346eb320, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:15:02,140-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [6dc7f21c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:15:02,141-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [6dc7f21c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [320f0eef] Autorecovering 2 hosts
>2018-05-29 09:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [320f0eef] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5645e60d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:20:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5645e60d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:20:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5645e60d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:20:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5645e60d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:20:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5645e60d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 9bb97b1
>2018-05-29 09:20:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5645e60d] FINISH, SetVdsStatusVDSCommand, log id: 9bb97b1
>2018-05-29 09:20:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5645e60d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:20:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5645e60d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:20:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5645e60d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:20:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5782d0e9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:20:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5782d0e9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:20:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5782d0e9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:20:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5782d0e9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:20:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5782d0e9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50ded7
>2018-05-29 09:20:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5782d0e9] FINISH, SetVdsStatusVDSCommand, log id: 50ded7
>2018-05-29 09:20:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [5782d0e9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3ee562d0
>2018-05-29 09:20:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [5782d0e9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3ee562d0
>2018-05-29 09:20:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5782d0e9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:20:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5782d0e9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:20:02,651-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [629dce32] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6e30584a
>2018-05-29 09:20:02,653-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [629dce32] FINISH, GetHardwareInfoVDSCommand, log id: 6e30584a
>2018-05-29 09:20:02,668-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [77ae512c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:20:02,676-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [47b2f001] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:20:02,681-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [77934311] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:20:02,739-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [77934311] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f0ee768
>2018-05-29 09:20:02,804-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [771be597] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3758af3d
>2018-05-29 09:20:02,806-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [771be597] FINISH, GetHardwareInfoVDSCommand, log id: 3758af3d
>2018-05-29 09:20:02,820-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [68114cb3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:20:02,825-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [19338e5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:20:02,830-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [53282368] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:20:02,837-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [53282368] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2e2189ce
>2018-05-29 09:20:03,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [53282368] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:20:03,226-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [53282368] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2e2189ce
>2018-05-29 09:20:03,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [53282368] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:20:03,231-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [53282368] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:20:03,235-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [53282368] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:20:03,236-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [53282368] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:20:03,239-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [53282368] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:20:03,240-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [53282368] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:20:03,243-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [53282368] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:20:03,246-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [53282368] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 53282368, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:20:03,252-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 71efa136
>2018-05-29 09:20:03,252-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:20:03,352-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:20:03,352-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:20:03,352-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:20:03,352-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:20:03,352-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 71efa136
>2018-05-29 09:20:03,352-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:20:03,366-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [73c14583] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:20:03,368-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73c14583] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6b09167b
>2018-05-29 09:20:03,370-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73c14583] FINISH, SetVdsStatusVDSCommand, log id: 6b09167b
>2018-05-29 09:20:03,401-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73c14583] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 73c14583, Job ID: 79172dde-18f2-42ec-b082-ad02daf7fef4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:20:03,406-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73c14583] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 19338e5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:20:03,409-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73c14583] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 19338e5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:20:03,420-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [7a229726] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:20:03,421-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7a229726] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:20:04,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [77934311] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:20:04,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [77934311] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6f0ee768
>2018-05-29 09:20:04,067-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [77934311] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:20:04,068-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [77934311] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:20:04,072-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [77934311] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:20:04,072-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [77934311] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:20:04,076-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [77934311] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:20:04,077-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [77934311] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:20:04,080-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [77934311] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:20:04,083-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [77934311] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 77934311, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:20:04,089-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 50d935fb
>2018-05-29 09:20:04,089-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:20:04,189-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:20:04,189-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:20:04,189-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:20:04,189-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:20:04,189-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 50d935fb
>2018-05-29 09:20:04,189-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:20:04,203-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [38404fb6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:20:04,204-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [38404fb6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5155feae
>2018-05-29 09:20:04,207-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [38404fb6] FINISH, SetVdsStatusVDSCommand, log id: 5155feae
>2018-05-29 09:20:04,242-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [38404fb6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 38404fb6, Job ID: 6a3a4296-f5e5-4060-81b6-dca7b10b0b6d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:20:04,247-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [38404fb6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 47b2f001, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:20:04,250-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [38404fb6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 47b2f001, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:20:04,261-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [3ae3e2a2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:20:04,262-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [3ae3e2a2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [75e8a26b] Autorecovering 2 hosts
>2018-05-29 09:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [75e8a26b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:25:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [568af21] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:25:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [568af21] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:25:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [568af21] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:25:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [568af21] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:25:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [568af21] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 22844149
>2018-05-29 09:25:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [568af21] FINISH, SetVdsStatusVDSCommand, log id: 22844149
>2018-05-29 09:25:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [568af21] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:25:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [568af21] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:25:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [568af21] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:25:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17e59f53] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:25:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17e59f53] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:25:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17e59f53] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:25:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17e59f53] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:25:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [17e59f53] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3c7346b
>2018-05-29 09:25:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [17e59f53] FINISH, SetVdsStatusVDSCommand, log id: 3c7346b
>2018-05-29 09:25:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [17e59f53] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 36511954
>2018-05-29 09:25:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [17e59f53] FINISH, SetHaMaintenanceModeVDSCommand, log id: 36511954
>2018-05-29 09:25:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17e59f53] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:25:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17e59f53] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:25:01,161-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [41fd8d39] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 590d2937
>2018-05-29 09:25:01,163-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [41fd8d39] FINISH, GetHardwareInfoVDSCommand, log id: 590d2937
>2018-05-29 09:25:01,175-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [96575fb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:25:01,184-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [41c01cba] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:25:01,189-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [449e5c22] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:25:01,200-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [449e5c22] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4ba862fe
>2018-05-29 09:25:01,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [449e5c22] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:25:01,566-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [449e5c22] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4ba862fe
>2018-05-29 09:25:01,570-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [449e5c22] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:25:01,571-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [449e5c22] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:25:01,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [449e5c22] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:25:01,576-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [449e5c22] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:25:01,580-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [449e5c22] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:25:01,580-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [449e5c22] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:25:01,584-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [449e5c22] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:25:01,586-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [449e5c22] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 449e5c22, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:25:01,590-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1083206c
>2018-05-29 09:25:01,590-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:25:01,686-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:25:01,686-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:25:01,686-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:25:01,686-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:25:01,686-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1083206c
>2018-05-29 09:25:01,686-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:25:01,699-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [76cdd52e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:25:01,700-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [76cdd52e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b622ebd
>2018-05-29 09:25:01,703-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [76cdd52e] FINISH, SetVdsStatusVDSCommand, log id: b622ebd
>2018-05-29 09:25:01,708-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [76cdd52e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 76cdd52e, Job ID: aa5a145d-2be2-4b25-a86d-b240688f6839, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:25:01,712-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [76cdd52e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 41c01cba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:25:01,715-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [76cdd52e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 41c01cba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:25:01,725-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [9933632] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:25:01,726-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [9933632] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:25:01,857-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2f709f64] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 26cd97c3
>2018-05-29 09:25:01,858-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2f709f64] FINISH, GetHardwareInfoVDSCommand, log id: 26cd97c3
>2018-05-29 09:25:01,871-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [415ba984] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:25:01,875-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [33fb95eb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:25:01,881-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1a1eab33] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:25:01,887-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1a1eab33] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 69ace64f
>2018-05-29 09:25:02,259-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1a1eab33] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:25:02,259-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1a1eab33] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 69ace64f
>2018-05-29 09:25:02,264-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1a1eab33] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:25:02,264-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1a1eab33] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:25:02,268-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1a1eab33] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:25:02,269-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1a1eab33] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:25:02,273-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1a1eab33] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:25:02,274-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1a1eab33] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:25:02,276-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1a1eab33] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:25:02,279-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1a1eab33] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1a1eab33, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:25:02,281-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 423b13ff
>2018-05-29 09:25:02,282-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:25:02,378-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:25:02,378-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:25:02,378-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:25:02,378-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:25:02,378-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 423b13ff
>2018-05-29 09:25:02,378-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:25:02,391-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [31e19f1d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:25:02,392-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [31e19f1d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c34111a
>2018-05-29 09:25:02,394-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [31e19f1d] FINISH, SetVdsStatusVDSCommand, log id: 2c34111a
>2018-05-29 09:25:02,400-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [31e19f1d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 31e19f1d, Job ID: c94737c2-62fa-4bd0-866f-468788afa215, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:25:02,405-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [31e19f1d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 33fb95eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:25:02,407-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [31e19f1d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 33fb95eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:25:02,417-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [15bb2508] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:25:02,418-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [15bb2508] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5b6c0a1c] Autorecovering 2 hosts
>2018-05-29 09:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5b6c0a1c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:30:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [770eec3a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:30:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [770eec3a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:30:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [770eec3a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:30:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [770eec3a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:30:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [770eec3a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2072b17d
>2018-05-29 09:30:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [770eec3a] FINISH, SetVdsStatusVDSCommand, log id: 2072b17d
>2018-05-29 09:30:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [770eec3a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:30:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [770eec3a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:30:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [770eec3a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:30:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [bf11c23] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:30:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [bf11c23] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:30:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [bf11c23] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:30:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [bf11c23] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:30:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [bf11c23] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13642be7
>2018-05-29 09:30:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [bf11c23] FINISH, SetVdsStatusVDSCommand, log id: 13642be7
>2018-05-29 09:30:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [bf11c23] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: ca2116b
>2018-05-29 09:30:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [bf11c23] FINISH, SetHaMaintenanceModeVDSCommand, log id: ca2116b
>2018-05-29 09:30:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [bf11c23] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:30:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [bf11c23] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:30:02,464-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [41c4890] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7c0d57a7
>2018-05-29 09:30:02,466-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [41c4890] FINISH, GetHardwareInfoVDSCommand, log id: 7c0d57a7
>2018-05-29 09:30:02,480-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [3b206289] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:30:02,489-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [379dc020] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:30:02,494-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [19e64136] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:30:02,506-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [19e64136] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7861be6
>2018-05-29 09:30:02,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [19e64136] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:30:02,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [19e64136] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7861be6
>2018-05-29 09:30:02,890-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [19e64136] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:30:02,890-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [19e64136] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:30:02,894-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [19e64136] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:30:02,895-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [19e64136] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:30:02,898-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [19e64136] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:30:02,899-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [19e64136] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:30:02,902-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [19e64136] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:30:02,904-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [19e64136] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 19e64136, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:30:02,910-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1abed62b
>2018-05-29 09:30:02,910-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:30:02,987-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [182520c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4cafb244
>2018-05-29 09:30:02,988-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [182520c] FINISH, GetHardwareInfoVDSCommand, log id: 4cafb244
>2018-05-29 09:30:03,001-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [71acd3d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:30:03,005-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [24e74bf9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:30:03,010-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [64f235d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:30:03,013-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:30:03,013-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:30:03,013-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:30:03,013-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:30:03,013-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1abed62b
>2018-05-29 09:30:03,013-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:30:03,023-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [64f235d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 497a0f6f
>2018-05-29 09:30:03,026-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [258ad0a3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:30:03,027-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [258ad0a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 207bc276
>2018-05-29 09:30:03,029-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [258ad0a3] FINISH, SetVdsStatusVDSCommand, log id: 207bc276
>2018-05-29 09:30:03,061-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [258ad0a3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 258ad0a3, Job ID: fd602033-6a7b-46c3-9bc2-4385645d6e43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:30:03,066-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [258ad0a3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 379dc020, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:30:03,068-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [258ad0a3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 379dc020, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:30:03,079-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [1e66b955] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:30:03,080-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [1e66b955] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:30:03,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [64f235d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:30:03,403-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [64f235d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 497a0f6f
>2018-05-29 09:30:03,407-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [64f235d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:30:03,407-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [64f235d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:30:03,411-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [64f235d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:30:03,412-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [64f235d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:30:03,416-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [64f235d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:30:03,416-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [64f235d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:30:03,419-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [64f235d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:30:03,422-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [64f235d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 64f235d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:30:03,428-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: e3f1761
>2018-05-29 09:30:03,428-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:30:03,528-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:30:03,528-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:30:03,528-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:30:03,528-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:30:03,528-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: e3f1761
>2018-05-29 09:30:03,528-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:30:03,541-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [e7bba15] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:30:03,543-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [e7bba15] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17a6e992
>2018-05-29 09:30:03,545-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [e7bba15] FINISH, SetVdsStatusVDSCommand, log id: 17a6e992
>2018-05-29 09:30:03,550-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [e7bba15] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: e7bba15, Job ID: 494fea00-a535-48e6-bfcd-7877a40933a4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:30:03,554-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [e7bba15] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 24e74bf9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:30:03,556-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [e7bba15] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 24e74bf9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:30:03,567-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [6c2919a7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:30:03,568-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [6c2919a7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [5fffab58] Autorecovering 2 hosts
>2018-05-29 09:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [5fffab58] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:35:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4ea4a581] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:35:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4ea4a581] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:35:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4ea4a581] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:35:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4ea4a581] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:35:00,051-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4ea4a581] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 448e3ad1
>2018-05-29 09:35:00,054-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4ea4a581] FINISH, SetVdsStatusVDSCommand, log id: 448e3ad1
>2018-05-29 09:35:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4ea4a581] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:35:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4ea4a581] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:35:00,057-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [4ea4a581] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:35:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4fe8a080] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4fe8a080] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4fe8a080] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4fe8a080] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:35:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4fe8a080] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: def8815
>2018-05-29 09:35:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4fe8a080] FINISH, SetVdsStatusVDSCommand, log id: def8815
>2018-05-29 09:35:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [4fe8a080] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6c1dfe77
>2018-05-29 09:35:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [4fe8a080] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6c1dfe77
>2018-05-29 09:35:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4fe8a080] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:35:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4fe8a080] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:35:00,857-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [6024a089] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 8b80e31
>2018-05-29 09:35:00,859-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [6024a089] FINISH, GetHardwareInfoVDSCommand, log id: 8b80e31
>2018-05-29 09:35:00,871-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [50beb0a2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:35:00,880-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [c0ab2b3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:35:00,885-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [15b350f5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:35:00,891-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [15b350f5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5253a57b
>2018-05-29 09:35:01,109-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [77c6e3a5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 23325434
>2018-05-29 09:35:01,110-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [77c6e3a5] FINISH, GetHardwareInfoVDSCommand, log id: 23325434
>2018-05-29 09:35:01,123-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [a54eae6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:35:01,127-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [7a7bdf30] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:35:01,133-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5f2a13ad] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:35:01,139-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5f2a13ad] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 76d6d10a
>2018-05-29 09:35:01,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5f2a13ad] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:35:01,517-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5f2a13ad] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 76d6d10a
>2018-05-29 09:35:01,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5f2a13ad] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:35:01,521-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5f2a13ad] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:35:01,525-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5f2a13ad] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:35:01,526-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5f2a13ad] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:35:01,530-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5f2a13ad] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:35:01,531-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5f2a13ad] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:35:01,534-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5f2a13ad] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:35:01,537-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5f2a13ad] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5f2a13ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:35:01,540-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 522aeeba
>2018-05-29 09:35:01,540-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:35:01,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:35:01,639-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:35:01,639-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:35:01,639-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:35:01,639-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 522aeeba
>2018-05-29 09:35:01,639-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:35:01,653-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2405b1fd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:35:01,655-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2405b1fd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 915e5b6
>2018-05-29 09:35:01,657-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2405b1fd] FINISH, SetVdsStatusVDSCommand, log id: 915e5b6
>2018-05-29 09:35:01,689-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2405b1fd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2405b1fd, Job ID: 3868a1b6-c557-43d4-bfc6-2ecd28c067c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:35:01,694-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2405b1fd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7a7bdf30, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:35:01,696-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2405b1fd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7a7bdf30, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:35:01,707-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [3cbeaaad] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:35:01,708-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [3cbeaaad] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:35:02,257-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15b350f5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:35:02,257-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [15b350f5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5253a57b
>2018-05-29 09:35:02,261-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15b350f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:35:02,262-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [15b350f5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:35:02,266-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15b350f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:35:02,267-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [15b350f5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:35:02,271-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15b350f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:35:02,271-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [15b350f5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:35:02,275-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [15b350f5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:35:02,277-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15b350f5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 15b350f5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:35:02,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 411974a6
>2018-05-29 09:35:02,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:35:02,376-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:35:02,376-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:35:02,376-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:35:02,376-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:35:02,376-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 411974a6
>2018-05-29 09:35:02,376-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:35:02,389-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [16112223] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:35:02,390-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [16112223] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7ee3c343
>2018-05-29 09:35:02,393-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [16112223] FINISH, SetVdsStatusVDSCommand, log id: 7ee3c343
>2018-05-29 09:35:02,398-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16112223] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 16112223, Job ID: b7b21f36-938f-48f4-983a-f300e0725b14, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:35:02,403-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16112223] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: c0ab2b3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:35:02,405-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16112223] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: c0ab2b3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:35:02,416-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [5f214e3c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:35:02,417-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [5f214e3c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [108ac6b4] Autorecovering 2 hosts
>2018-05-29 09:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [108ac6b4] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:40:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e6d7ebf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e6d7ebf] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e6d7ebf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e6d7ebf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:40:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6e6d7ebf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6247bcd
>2018-05-29 09:40:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6e6d7ebf] FINISH, SetVdsStatusVDSCommand, log id: 6247bcd
>2018-05-29 09:40:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e6d7ebf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:40:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e6d7ebf] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:40:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6e6d7ebf] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:40:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5ad589d1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:40:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5ad589d1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:40:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5ad589d1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:40:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5ad589d1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:40:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5ad589d1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 678c492c
>2018-05-29 09:40:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5ad589d1] FINISH, SetVdsStatusVDSCommand, log id: 678c492c
>2018-05-29 09:40:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [5ad589d1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 706b3e30
>2018-05-29 09:40:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [5ad589d1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 706b3e30
>2018-05-29 09:40:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5ad589d1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:40:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5ad589d1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:40:02,332-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1fbbc8df] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6f6e1b32
>2018-05-29 09:40:02,334-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1fbbc8df] FINISH, GetHardwareInfoVDSCommand, log id: 6f6e1b32
>2018-05-29 09:40:02,348-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [29cf889] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:40:02,356-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [7d982c93] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:40:02,362-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [41332d1f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:40:02,373-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [41332d1f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1f571aaf
>2018-05-29 09:40:02,750-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41332d1f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:40:02,750-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [41332d1f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1f571aaf
>2018-05-29 09:40:02,754-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41332d1f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:40:02,755-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41332d1f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:40:02,759-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41332d1f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:40:02,759-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41332d1f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:40:02,763-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41332d1f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:40:02,764-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41332d1f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:40:02,766-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [41332d1f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:40:02,769-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41332d1f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 41332d1f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:40:02,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5bd38a88
>2018-05-29 09:40:02,776-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:40:02,873-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:40:02,873-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:40:02,873-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:40:02,873-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:40:02,873-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5bd38a88
>2018-05-29 09:40:02,873-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:40:02,887-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [32e5856f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:40:02,888-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [32e5856f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 392f1cea
>2018-05-29 09:40:02,890-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [32e5856f] FINISH, SetVdsStatusVDSCommand, log id: 392f1cea
>2018-05-29 09:40:02,922-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [32e5856f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 32e5856f, Job ID: 04838009-cc03-4cf0-92b7-72994a9ea26c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:40:02,927-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [32e5856f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7d982c93, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:40:02,930-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [32e5856f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7d982c93, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:40:02,941-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [6b79fcb4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:40:02,942-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [6b79fcb4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:40:03,141-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2dc871a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7c07c03c
>2018-05-29 09:40:03,143-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2dc871a] FINISH, GetHardwareInfoVDSCommand, log id: 7c07c03c
>2018-05-29 09:40:03,157-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [b882535] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:40:03,161-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [4ebd025b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:40:03,167-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [b31a672] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:40:03,174-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [b31a672] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 220324f6
>2018-05-29 09:40:03,551-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [b31a672] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:40:03,551-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [b31a672] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 220324f6
>2018-05-29 09:40:03,555-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [b31a672] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:40:03,556-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [b31a672] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:40:03,560-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [b31a672] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:40:03,561-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [b31a672] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:40:03,565-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [b31a672] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:40:03,565-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [b31a672] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:40:03,568-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [b31a672] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:40:03,571-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [b31a672] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: b31a672, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:40:03,574-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2d35bcf
>2018-05-29 09:40:03,574-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:40:03,672-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:40:03,672-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:40:03,672-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:40:03,672-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:40:03,672-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2d35bcf
>2018-05-29 09:40:03,672-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:40:03,686-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [4d671fde] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:40:03,687-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4d671fde] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 212824ad
>2018-05-29 09:40:03,689-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4d671fde] FINISH, SetVdsStatusVDSCommand, log id: 212824ad
>2018-05-29 09:40:03,695-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4d671fde] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4d671fde, Job ID: 4c49a043-8360-432c-9b00-5f23136c57fe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:40:03,700-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4d671fde] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4ebd025b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:40:03,702-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4d671fde] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4ebd025b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:40:03,712-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [648e83cd] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:40:03,713-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [648e83cd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [66c0b3fc] Autorecovering 2 hosts
>2018-05-29 09:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [66c0b3fc] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:45:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [509052e1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [509052e1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [509052e1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [509052e1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:45:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [509052e1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3dc42f64
>2018-05-29 09:45:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [509052e1] FINISH, SetVdsStatusVDSCommand, log id: 3dc42f64
>2018-05-29 09:45:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [509052e1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:45:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [509052e1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:45:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [509052e1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:45:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [40036a4e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:45:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [40036a4e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:45:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [40036a4e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:45:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [40036a4e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:45:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [40036a4e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3439c1b2
>2018-05-29 09:45:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [40036a4e] FINISH, SetVdsStatusVDSCommand, log id: 3439c1b2
>2018-05-29 09:45:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [40036a4e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 51f04703
>2018-05-29 09:45:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [40036a4e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 51f04703
>2018-05-29 09:45:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [40036a4e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:45:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [40036a4e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:45:00,512-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [679a22e4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 893c3c3
>2018-05-29 09:45:00,514-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [679a22e4] FINISH, GetHardwareInfoVDSCommand, log id: 893c3c3
>2018-05-29 09:45:00,528-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [2bd68be4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:45:00,537-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [17b37e34] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:45:00,542-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3fe9e097] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:45:00,553-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3fe9e097] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1b16df26
>2018-05-29 09:45:00,931-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3fe9e097] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:45:00,931-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3fe9e097] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1b16df26
>2018-05-29 09:45:00,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3fe9e097] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:45:00,936-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3fe9e097] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:45:00,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3fe9e097] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:45:00,941-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3fe9e097] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:45:00,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3fe9e097] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:45:00,945-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3fe9e097] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:45:00,948-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3fe9e097] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:45:00,951-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3fe9e097] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3fe9e097, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:45:00,957-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3b888fae
>2018-05-29 09:45:00,957-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:45:01,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:45:01,056-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:45:01,056-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:45:01,056-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:45:01,056-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3b888fae
>2018-05-29 09:45:01,056-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:45:01,070-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [30a614b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:45:01,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [30a614b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5c333300
>2018-05-29 09:45:01,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [30a614b] FINISH, SetVdsStatusVDSCommand, log id: 5c333300
>2018-05-29 09:45:01,105-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [30a614b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 30a614b, Job ID: 7eb63609-24d7-466e-a1c1-fc34aef075fd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:45:01,110-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [30a614b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17b37e34, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:45:01,112-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [30a614b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17b37e34, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:45:01,123-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [29c6e35a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:45:01,124-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [29c6e35a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:45:01,471-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3f01b117] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 38e51678
>2018-05-29 09:45:01,473-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3f01b117] FINISH, GetHardwareInfoVDSCommand, log id: 38e51678
>2018-05-29 09:45:01,485-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [598b3def] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:45:01,490-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [288d0f25] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:45:01,496-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1e154fed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:45:01,503-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1e154fed] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6311153c
>2018-05-29 09:45:01,869-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e154fed] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:45:01,869-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1e154fed] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6311153c
>2018-05-29 09:45:01,873-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e154fed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:45:01,874-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e154fed] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:45:01,878-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e154fed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:45:01,878-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e154fed] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:45:01,882-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e154fed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:45:01,883-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e154fed] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:45:01,886-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1e154fed] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:45:01,889-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e154fed] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1e154fed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:45:01,892-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6ec36dbe
>2018-05-29 09:45:01,892-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:45:01,988-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:45:01,988-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:45:01,988-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:45:01,988-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:45:01,988-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6ec36dbe
>2018-05-29 09:45:01,988-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:45:02,001-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [7df81419] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:45:02,002-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7df81419] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e81a704
>2018-05-29 09:45:02,004-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7df81419] FINISH, SetVdsStatusVDSCommand, log id: 5e81a704
>2018-05-29 09:45:02,010-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7df81419] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7df81419, Job ID: 2e77edc1-6f5c-4870-8474-17444dea182f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:45:02,015-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7df81419] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 288d0f25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:45:02,017-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7df81419] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 288d0f25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:45:02,027-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [2804112d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:45:02,028-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [2804112d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4a2cf1e] Autorecovering 2 hosts
>2018-05-29 09:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4a2cf1e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:50:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [426f77a7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:50:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [426f77a7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:50:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [426f77a7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:50:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [426f77a7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:50:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [426f77a7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 14e3e2c0
>2018-05-29 09:50:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [426f77a7] FINISH, SetVdsStatusVDSCommand, log id: 14e3e2c0
>2018-05-29 09:50:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [426f77a7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:50:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [426f77a7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:50:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [426f77a7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:50:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [25b3d09f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:50:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [25b3d09f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:50:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [25b3d09f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:50:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [25b3d09f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:50:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [25b3d09f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53d64cca
>2018-05-29 09:50:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [25b3d09f] FINISH, SetVdsStatusVDSCommand, log id: 53d64cca
>2018-05-29 09:50:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [25b3d09f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1f82aaf5
>2018-05-29 09:50:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [25b3d09f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1f82aaf5
>2018-05-29 09:50:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [25b3d09f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:50:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [25b3d09f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:50:01,713-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4ce0f7ce] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 78fa9a4a
>2018-05-29 09:50:01,714-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4ce0f7ce] FINISH, GetHardwareInfoVDSCommand, log id: 78fa9a4a
>2018-05-29 09:50:01,727-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [7285f44c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:50:01,736-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [52b1a238] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:50:01,741-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [5431bb69] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:50:01,752-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [5431bb69] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5056720d
>2018-05-29 09:50:02,132-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5431bb69] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:50:02,132-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [5431bb69] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5056720d
>2018-05-29 09:50:02,136-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5431bb69] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:50:02,137-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5431bb69] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:50:02,140-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5431bb69] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:50:02,141-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5431bb69] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:50:02,145-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5431bb69] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:50:02,146-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5431bb69] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:50:02,149-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [5431bb69] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:50:02,152-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5431bb69] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5431bb69, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:50:02,159-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 50ee785c
>2018-05-29 09:50:02,159-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:50:02,259-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:50:02,259-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:50:02,259-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:50:02,259-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:50:02,259-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 50ee785c
>2018-05-29 09:50:02,259-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:50:02,272-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [612d0728] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:50:02,273-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [612d0728] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3955d8ad
>2018-05-29 09:50:02,276-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [612d0728] FINISH, SetVdsStatusVDSCommand, log id: 3955d8ad
>2018-05-29 09:50:02,308-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [612d0728] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 612d0728, Job ID: 714fc5ff-d205-4571-9b51-7772ff2d933a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:50:02,314-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [612d0728] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 52b1a238, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:50:02,316-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [612d0728] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 52b1a238, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:50:02,328-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [3980f41a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:50:02,329-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [3980f41a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:50:02,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4f4b683e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 49984e73
>2018-05-29 09:50:02,754-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4f4b683e] FINISH, GetHardwareInfoVDSCommand, log id: 49984e73
>2018-05-29 09:50:02,767-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [738e0835] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:50:02,772-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [15e35023] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:50:02,777-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [290e55c4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:50:02,784-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [290e55c4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 25b06092
>2018-05-29 09:50:03,154-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [290e55c4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:50:03,154-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [290e55c4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 25b06092
>2018-05-29 09:50:03,158-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [290e55c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:50:03,159-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [290e55c4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:50:03,163-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [290e55c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:50:03,164-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [290e55c4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:50:03,168-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [290e55c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:50:03,168-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [290e55c4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:50:03,172-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [290e55c4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:50:03,174-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [290e55c4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 290e55c4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:50:03,177-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4516e62d
>2018-05-29 09:50:03,177-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:50:03,274-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:50:03,274-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:50:03,274-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:50:03,274-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:50:03,275-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4516e62d
>2018-05-29 09:50:03,275-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:50:03,288-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [94f160d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:50:03,289-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [94f160d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3fdbdafa
>2018-05-29 09:50:03,291-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [94f160d] FINISH, SetVdsStatusVDSCommand, log id: 3fdbdafa
>2018-05-29 09:50:03,297-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [94f160d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 94f160d, Job ID: c76ec90f-134a-4a9d-aa4d-e4167b4b8d48, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:50:03,301-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [94f160d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 15e35023, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:50:03,303-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [94f160d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 15e35023, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:50:03,313-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [49374cc8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:50:03,314-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [49374cc8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:54:42,979-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler2) [3a649785] Attempting to update VMs/Templates Ovf.
>2018-05-29 09:54:42,982-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 09:54:42,982-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 09:54:42,983-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:54:42,987-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 09:54:42,990-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 09:54:42,990-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 09:54:42,990-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 09:54:42,990-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 09:54:42,991-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 09:54:42,991-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 09:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [a8fd26e] Autorecovering 2 hosts
>2018-05-29 09:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [a8fd26e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:55:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [161ac0c6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [161ac0c6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [161ac0c6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [161ac0c6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:55:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [161ac0c6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53a532a
>2018-05-29 09:55:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [161ac0c6] FINISH, SetVdsStatusVDSCommand, log id: 53a532a
>2018-05-29 09:55:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [161ac0c6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:55:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [161ac0c6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 09:55:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [161ac0c6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:55:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2740f644] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:55:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2740f644] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 09:55:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2740f644] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:55:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2740f644] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:55:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2740f644] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 703b1f74
>2018-05-29 09:55:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2740f644] FINISH, SetVdsStatusVDSCommand, log id: 703b1f74
>2018-05-29 09:55:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [2740f644] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 10ea5340
>2018-05-29 09:55:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [2740f644] FINISH, SetHaMaintenanceModeVDSCommand, log id: 10ea5340
>2018-05-29 09:55:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2740f644] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 09:55:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2740f644] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 09:55:01,011-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1d97d98] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5d09e951
>2018-05-29 09:55:01,013-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1d97d98] FINISH, GetHardwareInfoVDSCommand, log id: 5d09e951
>2018-05-29 09:55:01,026-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [40135cb2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:55:01,035-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1007512b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:55:01,040-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [68a86e5c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:55:01,051-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [68a86e5c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 355b05d1
>2018-05-29 09:55:02,428-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [68a86e5c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:55:02,428-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [68a86e5c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 355b05d1
>2018-05-29 09:55:02,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [68a86e5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:55:02,433-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [68a86e5c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:55:02,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [68a86e5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:55:02,437-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [68a86e5c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:55:02,441-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [68a86e5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:55:02,442-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [68a86e5c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:55:02,445-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [68a86e5c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:55:02,447-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [68a86e5c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 68a86e5c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:55:02,454-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7a89c17b
>2018-05-29 09:55:02,454-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:55:02,551-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:55:02,551-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:55:02,551-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:55:02,551-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:55:02,551-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7a89c17b
>2018-05-29 09:55:02,552-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:55:02,564-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [36ee077b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:55:02,566-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [36ee077b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a2cf2bd
>2018-05-29 09:55:02,568-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [36ee077b] FINISH, SetVdsStatusVDSCommand, log id: 5a2cf2bd
>2018-05-29 09:55:02,598-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [36ee077b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 36ee077b, Job ID: 3d3d6690-93f3-49a8-b603-5e717fe3730f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:55:02,603-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [36ee077b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1007512b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:55:02,605-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [36ee077b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1007512b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:55:02,618-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [7d26fe1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 09:55:02,619-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [7d26fe1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 09:55:03,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5c8c1231] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 138a81e0
>2018-05-29 09:55:03,074-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5c8c1231] FINISH, GetHardwareInfoVDSCommand, log id: 138a81e0
>2018-05-29 09:55:03,087-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [42ec113e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:55:03,091-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [1462f5f0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:55:03,097-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [737120b2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 09:55:03,103-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [737120b2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4642d425
>2018-05-29 09:55:03,476-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [737120b2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 09:55:03,476-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [737120b2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4642d425
>2018-05-29 09:55:03,480-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [737120b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:55:03,481-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [737120b2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:55:03,485-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [737120b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:55:03,485-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [737120b2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:55:03,489-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [737120b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 09:55:03,490-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [737120b2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 09:55:03,493-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [737120b2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 09:55:03,496-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [737120b2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 737120b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 09:55:03,499-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5ba1ffef
>2018-05-29 09:55:03,499-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 09:55:03,594-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:55:03,594-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 09:55:03,594-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 09:55:03,594-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 09:55:03,594-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5ba1ffef
>2018-05-29 09:55:03,594-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 09:55:03,607-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [d48853] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:55:03,608-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [d48853] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1711f144
>2018-05-29 09:55:03,610-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [d48853] FINISH, SetVdsStatusVDSCommand, log id: 1711f144
>2018-05-29 09:55:03,616-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [d48853] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: d48853, Job ID: fe434561-98b0-4494-b38e-ac8a468229f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 09:55:03,621-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [d48853] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1462f5f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 09:55:03,623-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [d48853] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1462f5f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 09:55:03,634-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [1a16ac23] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 09:55:03,635-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [1a16ac23] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:00:00,016-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [50386b8] Autorecovering 2 hosts
>2018-05-29 10:00:00,016-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [50386b8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:00:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [26ce2045] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:00:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [26ce2045] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:00:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [26ce2045] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:00:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [26ce2045] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:00:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [26ce2045] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 125309d2
>2018-05-29 10:00:01,918-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [26ce2045] FINISH, SetVdsStatusVDSCommand, log id: 125309d2
>2018-05-29 10:00:01,919-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [26ce2045] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:00:01,921-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [26ce2045] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:00:01,921-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [26ce2045] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:00:01,927-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [481b2b59] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:00:01,933-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [481b2b59] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:00:01,933-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [481b2b59] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:00:01,933-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [481b2b59] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:00:01,935-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [481b2b59] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8dab266
>2018-05-29 10:00:01,937-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [481b2b59] FINISH, SetVdsStatusVDSCommand, log id: 8dab266
>2018-05-29 10:00:01,940-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [481b2b59] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 729c56b7
>2018-05-29 10:00:01,942-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [481b2b59] FINISH, SetHaMaintenanceModeVDSCommand, log id: 729c56b7
>2018-05-29 10:00:01,942-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [481b2b59] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:00:01,943-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [481b2b59] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:00:03,226-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [34c79e2c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 76a2eded
>2018-05-29 10:00:03,228-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [34c79e2c] FINISH, GetHardwareInfoVDSCommand, log id: 76a2eded
>2018-05-29 10:00:03,242-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [2de44eb9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:00:03,251-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [742b69f4] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:00:03,256-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51949ee1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:00:03,263-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51949ee1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5df3df0d
>2018-05-29 10:00:03,641-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51949ee1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:00:03,641-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51949ee1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5df3df0d
>2018-05-29 10:00:03,645-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51949ee1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:00:03,646-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51949ee1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:00:03,650-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51949ee1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:00:03,651-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51949ee1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:00:03,655-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51949ee1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:00:03,655-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51949ee1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:00:03,660-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51949ee1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:00:03,662-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51949ee1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 51949ee1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:00:03,665-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 761d2cda
>2018-05-29 10:00:03,665-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:00:03,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:00:03,767-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:00:03,767-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:00:03,767-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:00:03,767-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 761d2cda
>2018-05-29 10:00:03,767-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:00:03,780-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5d56d8a3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:00:03,782-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5d56d8a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2f3ce39d
>2018-05-29 10:00:03,784-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5d56d8a3] FINISH, SetVdsStatusVDSCommand, log id: 2f3ce39d
>2018-05-29 10:00:03,816-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5d56d8a3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5d56d8a3, Job ID: b687a74d-4524-4444-aa31-ce7cf1535463, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:00:03,821-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5d56d8a3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 742b69f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:00:03,824-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5d56d8a3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 742b69f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:00:03,836-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [34acc635] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:00:03,837-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [34acc635] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:00:04,379-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [65b9de25] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: c96c8f8
>2018-05-29 10:00:04,381-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [65b9de25] FINISH, GetHardwareInfoVDSCommand, log id: c96c8f8
>2018-05-29 10:00:04,395-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [5e902ba2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:00:04,404-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [6c0bf229] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:00:04,409-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [79d03b8d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:00:04,420-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [79d03b8d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2324a1e7
>2018-05-29 10:00:04,800-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [79d03b8d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:00:04,800-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [79d03b8d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2324a1e7
>2018-05-29 10:00:04,804-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [79d03b8d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:00:04,805-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [79d03b8d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:00:04,809-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [79d03b8d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:00:04,810-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [79d03b8d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:00:04,813-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [79d03b8d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:00:04,814-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [79d03b8d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:00:04,817-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [79d03b8d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:00:04,820-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [79d03b8d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 79d03b8d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:00:04,826-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 755701f3
>2018-05-29 10:00:04,826-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:00:04,927-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:00:04,927-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:00:04,927-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:00:04,927-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:00:04,927-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 755701f3
>2018-05-29 10:00:04,927-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:00:04,941-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [3f39258] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:00:04,942-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3f39258] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2456ddf7
>2018-05-29 10:00:04,945-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3f39258] FINISH, SetVdsStatusVDSCommand, log id: 2456ddf7
>2018-05-29 10:00:04,977-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3f39258] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3f39258, Job ID: 75a6e45c-41dd-4082-a8d7-06973e73f8c7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:00:04,982-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3f39258] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6c0bf229, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:00:04,984-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3f39258] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6c0bf229, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:00:04,996-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [1fc5d342] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:00:04,997-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1fc5d342] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [79e3fa26] Autorecovering 2 hosts
>2018-05-29 10:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [79e3fa26] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:05:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7c6f662e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:05:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7c6f662e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7c6f662e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7c6f662e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:05:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7c6f662e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b898196
>2018-05-29 10:05:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7c6f662e] FINISH, SetVdsStatusVDSCommand, log id: 3b898196
>2018-05-29 10:05:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7c6f662e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:05:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7c6f662e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:05:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7c6f662e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:05:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [54b27274] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [54b27274] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [54b27274] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [54b27274] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:05:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [54b27274] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6767eea4
>2018-05-29 10:05:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [54b27274] FINISH, SetVdsStatusVDSCommand, log id: 6767eea4
>2018-05-29 10:05:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [54b27274] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 75a37b5e
>2018-05-29 10:05:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [54b27274] FINISH, SetHaMaintenanceModeVDSCommand, log id: 75a37b5e
>2018-05-29 10:05:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [54b27274] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:05:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [54b27274] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:05:01,464-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [7f8db494] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 66895f80
>2018-05-29 10:05:01,466-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [7f8db494] FINISH, GetHardwareInfoVDSCommand, log id: 66895f80
>2018-05-29 10:05:01,479-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [12171c1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:05:01,483-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3e31b0cb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:05:01,489-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7ed4e602] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:05:01,500-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7ed4e602] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 70c036f3
>2018-05-29 10:05:02,658-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [6e551734] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4b4e1d0e
>2018-05-29 10:05:02,660-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [6e551734] FINISH, GetHardwareInfoVDSCommand, log id: 4b4e1d0e
>2018-05-29 10:05:02,673-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [5a885d2f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:05:02,678-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6f78221b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:05:02,683-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [363eb744] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:05:02,690-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [363eb744] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 42011188
>2018-05-29 10:05:02,872-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ed4e602] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:05:02,872-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7ed4e602] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 70c036f3
>2018-05-29 10:05:02,876-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ed4e602] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:05:02,877-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7ed4e602] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:05:02,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ed4e602] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:05:02,881-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7ed4e602] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:05:02,885-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ed4e602] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:05:02,886-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7ed4e602] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:05:02,888-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7ed4e602] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:05:02,891-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ed4e602] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7ed4e602, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:05:02,898-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6707c711
>2018-05-29 10:05:02,898-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:05:02,998-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:05:02,998-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:05:02,998-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:05:02,998-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:05:02,998-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6707c711
>2018-05-29 10:05:02,998-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:05:03,011-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [58bcc1a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:05:03,012-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [58bcc1a6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1cd9314d
>2018-05-29 10:05:03,014-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [58bcc1a6] FINISH, SetVdsStatusVDSCommand, log id: 1cd9314d
>2018-05-29 10:05:03,022-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58bcc1a6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 58bcc1a6, Job ID: 09ee1422-77a9-41cb-a1f2-a9df21a84afc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:05:03,026-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58bcc1a6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3e31b0cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:05:03,029-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58bcc1a6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3e31b0cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:05:03,039-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [2b7c2245] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:05:03,040-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2b7c2245] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:05:03,075-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [363eb744] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:05:03,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [363eb744] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 42011188
>2018-05-29 10:05:03,079-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [363eb744] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:05:03,080-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [363eb744] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:05:03,084-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [363eb744] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:05:03,085-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [363eb744] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:05:03,088-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [363eb744] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:05:03,089-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [363eb744] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:05:03,092-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [363eb744] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:05:03,095-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [363eb744] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 363eb744, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:05:03,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a09aa69
>2018-05-29 10:05:03,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:05:03,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:05:03,198-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:05:03,198-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:05:03,198-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:05:03,198-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a09aa69
>2018-05-29 10:05:03,198-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:05:03,212-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1c5fc324] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:05:03,213-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1c5fc324] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ee42aca
>2018-05-29 10:05:03,215-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1c5fc324] FINISH, SetVdsStatusVDSCommand, log id: 6ee42aca
>2018-05-29 10:05:03,220-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c5fc324] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1c5fc324, Job ID: 42336737-a2aa-45cb-ab1f-9be459ccfe18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:05:03,225-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c5fc324] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6f78221b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:05:03,227-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c5fc324] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6f78221b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:05:03,237-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [2b4dd028] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:05:03,239-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [2b4dd028] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [51b4b1da] Autorecovering 2 hosts
>2018-05-29 10:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [51b4b1da] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:10:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18a16cff] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18a16cff] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18a16cff] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18a16cff] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:10:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [18a16cff] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b9a7e5d
>2018-05-29 10:10:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [18a16cff] FINISH, SetVdsStatusVDSCommand, log id: 3b9a7e5d
>2018-05-29 10:10:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18a16cff] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:10:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18a16cff] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:10:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [18a16cff] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:10:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [67605445] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:10:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [67605445] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [67605445] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [67605445] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:10:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [67605445] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6b40fa2
>2018-05-29 10:10:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [67605445] FINISH, SetVdsStatusVDSCommand, log id: 6b40fa2
>2018-05-29 10:10:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [67605445] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6cdbf39b
>2018-05-29 10:10:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [67605445] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6cdbf39b
>2018-05-29 10:10:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [67605445] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:10:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [67605445] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:10:01,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [de132eb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 524de3d
>2018-05-29 10:10:01,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [de132eb] FINISH, GetHardwareInfoVDSCommand, log id: 524de3d
>2018-05-29 10:10:01,091-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [1c64338] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:10:01,099-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [70aefe3b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:10:01,104-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [7427da44] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:10:01,116-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [7427da44] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1ab9964c
>2018-05-29 10:10:01,164-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1079033a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5937d40c
>2018-05-29 10:10:01,165-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1079033a] FINISH, GetHardwareInfoVDSCommand, log id: 5937d40c
>2018-05-29 10:10:01,179-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [79ffb4bd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:10:01,183-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [5dc90903] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:10:01,189-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [58dde58f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:10:01,195-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [58dde58f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 65df76d8
>2018-05-29 10:10:01,495-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7427da44] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:10:01,495-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [7427da44] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1ab9964c
>2018-05-29 10:10:01,499-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7427da44] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:10:01,500-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7427da44] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:10:01,504-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7427da44] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:10:01,504-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7427da44] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:10:01,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7427da44] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:10:01,509-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7427da44] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:10:01,512-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [7427da44] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:10:01,515-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7427da44] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7427da44, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:10:01,521-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7199db5a
>2018-05-29 10:10:01,521-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:10:01,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [58dde58f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:10:01,575-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [58dde58f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 65df76d8
>2018-05-29 10:10:01,579-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [58dde58f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:10:01,579-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [58dde58f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:10:01,583-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [58dde58f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:10:01,584-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [58dde58f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:10:01,588-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [58dde58f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:10:01,588-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [58dde58f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:10:01,591-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [58dde58f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:10:01,594-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [58dde58f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 58dde58f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:10:01,620-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:10:01,620-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:10:01,620-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:10:01,620-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:10:01,620-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7199db5a
>2018-05-29 10:10:01,620-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:10:01,622-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6234ce0c
>2018-05-29 10:10:01,622-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:10:01,634-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [6b5903c1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:10:01,635-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6b5903c1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41c12090
>2018-05-29 10:10:01,637-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6b5903c1] FINISH, SetVdsStatusVDSCommand, log id: 41c12090
>2018-05-29 10:10:01,671-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6b5903c1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6b5903c1, Job ID: b84fd694-47e1-4ffc-9e9f-b8813d6aa13e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:10:01,675-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6b5903c1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 70aefe3b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:10:01,678-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6b5903c1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 70aefe3b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:10:01,689-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [6f68a894] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:10:01,690-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [6f68a894] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:10:01,723-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:10:01,723-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:10:01,723-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:10:01,723-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:10:01,723-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6234ce0c
>2018-05-29 10:10:01,723-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:10:01,736-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [1e4fe1f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:10:01,737-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1e4fe1f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 608ea763
>2018-05-29 10:10:01,739-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1e4fe1f] FINISH, SetVdsStatusVDSCommand, log id: 608ea763
>2018-05-29 10:10:01,745-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1e4fe1f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1e4fe1f, Job ID: 50a2927b-0fcf-4cda-b057-29e6cb3a1fc1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:10:01,750-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1e4fe1f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5dc90903, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:10:01,752-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1e4fe1f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5dc90903, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:10:01,762-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [44366838] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:10:01,763-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [44366838] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4406d76e] Autorecovering 2 hosts
>2018-05-29 10:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4406d76e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:15:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4260da72] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:15:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4260da72] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4260da72] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4260da72] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:15:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4260da72] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72a2a115
>2018-05-29 10:15:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4260da72] FINISH, SetVdsStatusVDSCommand, log id: 72a2a115
>2018-05-29 10:15:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4260da72] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:15:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4260da72] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:15:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4260da72] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:15:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4612a7ab] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:15:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4612a7ab] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:15:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4612a7ab] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:15:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4612a7ab] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:15:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4612a7ab] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2966256f
>2018-05-29 10:15:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4612a7ab] FINISH, SetVdsStatusVDSCommand, log id: 2966256f
>2018-05-29 10:15:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4612a7ab] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 530275b6
>2018-05-29 10:15:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4612a7ab] FINISH, SetHaMaintenanceModeVDSCommand, log id: 530275b6
>2018-05-29 10:15:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4612a7ab] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:15:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4612a7ab] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:15:02,504-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [6350b44a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 73d50e5c
>2018-05-29 10:15:02,505-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [6350b44a] FINISH, GetHardwareInfoVDSCommand, log id: 73d50e5c
>2018-05-29 10:15:02,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [41d0cf31] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 44993ca6
>2018-05-29 10:15:02,514-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [41d0cf31] FINISH, GetHardwareInfoVDSCommand, log id: 44993ca6
>2018-05-29 10:15:02,521-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [1b7ec227] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:15:02,527-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [14594626] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:15:02,530-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [68cfc8b4] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:15:02,536-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [434c6540] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:15:02,536-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [17119ced] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:15:02,542-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4e17d5dd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:15:02,549-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [434c6540] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 568c335
>2018-05-29 10:15:02,554-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4e17d5dd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 683e814b
>2018-05-29 10:15:02,925-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4e17d5dd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:15:02,925-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4e17d5dd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 683e814b
>2018-05-29 10:15:02,926-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [434c6540] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:15:02,926-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [434c6540] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 568c335
>2018-05-29 10:15:02,929-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4e17d5dd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:15:02,929-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4e17d5dd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:15:02,930-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [434c6540] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:15:02,931-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [434c6540] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:15:02,934-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4e17d5dd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:15:02,934-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4e17d5dd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:15:02,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [434c6540] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:15:02,936-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [434c6540] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:15:02,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4e17d5dd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:15:02,939-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4e17d5dd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:15:02,939-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [434c6540] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:15:02,940-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [434c6540] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:15:02,942-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4e17d5dd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:15:02,944-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [434c6540] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:15:02,945-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4e17d5dd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4e17d5dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:15:02,946-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [434c6540] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 434c6540, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:15:02,952-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2589e26e
>2018-05-29 10:15:02,952-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:15:03,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:15:03,048-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:15:03,048-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:15:03,048-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:15:03,048-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2589e26e
>2018-05-29 10:15:03,048-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:15:03,049-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 50ba0d13
>2018-05-29 10:15:03,049-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:15:03,062-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [102d9ca0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:15:03,063-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [102d9ca0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17b4b4db
>2018-05-29 10:15:03,065-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [102d9ca0] FINISH, SetVdsStatusVDSCommand, log id: 17b4b4db
>2018-05-29 10:15:03,098-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [102d9ca0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 102d9ca0, Job ID: 9fcfb01c-43cb-40c8-a303-9f08f0934698, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:15:03,103-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [102d9ca0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17119ced, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:15:03,105-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [102d9ca0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17119ced, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:15:03,116-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [292b573d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:15:03,117-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [292b573d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:15:03,151-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:15:03,151-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:15:03,151-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:15:03,151-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:15:03,151-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 50ba0d13
>2018-05-29 10:15:03,152-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:15:03,164-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [3f821d9b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:15:03,165-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3f821d9b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: c132f1e
>2018-05-29 10:15:03,167-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3f821d9b] FINISH, SetVdsStatusVDSCommand, log id: c132f1e
>2018-05-29 10:15:03,173-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3f821d9b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3f821d9b, Job ID: b4d03afb-4e81-41f3-8f23-4adcb80dd391, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:15:03,177-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3f821d9b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 68cfc8b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:15:03,180-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3f821d9b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 68cfc8b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:15:03,190-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [3f5fdf35] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:15:03,191-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [3f5fdf35] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [3b702f0d] Autorecovering 2 hosts
>2018-05-29 10:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [3b702f0d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:20:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [56821dc5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:20:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [56821dc5] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:20:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [56821dc5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:20:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [56821dc5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:20:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [56821dc5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 675cc73e
>2018-05-29 10:20:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [56821dc5] FINISH, SetVdsStatusVDSCommand, log id: 675cc73e
>2018-05-29 10:20:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [56821dc5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:20:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [56821dc5] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:20:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [56821dc5] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:20:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5a103525] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:20:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5a103525] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:20:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5a103525] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:20:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5a103525] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:20:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5a103525] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1d32e639
>2018-05-29 10:20:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5a103525] FINISH, SetVdsStatusVDSCommand, log id: 1d32e639
>2018-05-29 10:20:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5a103525] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3ca9abde
>2018-05-29 10:20:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5a103525] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3ca9abde
>2018-05-29 10:20:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5a103525] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:20:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5a103525] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:20:00,750-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3c037293] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 42ecfa97
>2018-05-29 10:20:00,751-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [69343b5c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 236bec59
>2018-05-29 10:20:00,752-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3c037293] FINISH, GetHardwareInfoVDSCommand, log id: 42ecfa97
>2018-05-29 10:20:00,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [69343b5c] FINISH, GetHardwareInfoVDSCommand, log id: 236bec59
>2018-05-29 10:20:00,765-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [6696dff4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:20:00,766-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [731413b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:20:00,770-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [170b3cb9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:20:00,775-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7243a8bb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:20:00,775-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [54e09641] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:20:00,785-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5780f647] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:20:00,792-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7243a8bb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3b2cccec
>2018-05-29 10:20:00,840-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5780f647] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 784f5dd5
>2018-05-29 10:20:01,162-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7243a8bb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:20:01,162-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7243a8bb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3b2cccec
>2018-05-29 10:20:01,166-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7243a8bb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:20:01,166-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7243a8bb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:20:01,170-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7243a8bb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:20:01,171-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7243a8bb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:20:01,174-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7243a8bb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:20:01,175-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7243a8bb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:20:01,178-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7243a8bb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:20:01,180-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7243a8bb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7243a8bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:20:01,186-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 642060dd
>2018-05-29 10:20:01,186-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:20:01,216-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5780f647] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:20:01,216-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5780f647] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 784f5dd5
>2018-05-29 10:20:01,220-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5780f647] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:20:01,221-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5780f647] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:20:01,225-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5780f647] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:20:01,226-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5780f647] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:20:01,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5780f647] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:20:01,230-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5780f647] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:20:01,233-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5780f647] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:20:01,236-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5780f647] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5780f647, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:20:01,284-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:20:01,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:20:01,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:20:01,285-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:20:01,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 642060dd
>2018-05-29 10:20:01,285-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:20:01,286-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 11480024
>2018-05-29 10:20:01,286-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:20:01,298-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [1208e14c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:20:01,299-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1208e14c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 113b79d7
>2018-05-29 10:20:01,301-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1208e14c] FINISH, SetVdsStatusVDSCommand, log id: 113b79d7
>2018-05-29 10:20:01,310-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1208e14c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1208e14c, Job ID: 6e33866c-9b36-4498-92cf-4e82f8703222, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:20:01,315-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1208e14c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 170b3cb9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:20:01,317-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1208e14c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 170b3cb9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:20:01,327-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [17609217] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:20:01,328-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [17609217] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:20:01,382-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:20:01,382-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:20:01,382-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:20:01,382-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:20:01,382-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 11480024
>2018-05-29 10:20:01,382-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:20:01,394-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [58282f82] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:20:01,396-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [58282f82] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4f608ec4
>2018-05-29 10:20:01,397-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [58282f82] FINISH, SetVdsStatusVDSCommand, log id: 4f608ec4
>2018-05-29 10:20:01,403-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [58282f82] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 58282f82, Job ID: 6426a1a7-fc6c-4d20-b935-e844e71093c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:20:01,407-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [58282f82] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 54e09641, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:20:01,409-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [58282f82] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 54e09641, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:20:01,420-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [3290ba7a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:20:01,421-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [3290ba7a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6472d5e6] Autorecovering 2 hosts
>2018-05-29 10:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6472d5e6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:25:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [d96f7ec] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:25:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [d96f7ec] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:25:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [d96f7ec] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:25:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [d96f7ec] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:25:00,050-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [d96f7ec] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ed9c962
>2018-05-29 10:25:00,052-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [d96f7ec] FINISH, SetVdsStatusVDSCommand, log id: ed9c962
>2018-05-29 10:25:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [d96f7ec] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:25:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [d96f7ec] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:25:00,055-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [d96f7ec] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:25:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2638ff3d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:25:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2638ff3d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:25:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2638ff3d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:25:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2638ff3d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:25:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2638ff3d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 180d91e1
>2018-05-29 10:25:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2638ff3d] FINISH, SetVdsStatusVDSCommand, log id: 180d91e1
>2018-05-29 10:25:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [2638ff3d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 37b6dca
>2018-05-29 10:25:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [2638ff3d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 37b6dca
>2018-05-29 10:25:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2638ff3d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:25:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2638ff3d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:25:02,089-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [24304e97] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 248c0dfc
>2018-05-29 10:25:02,090-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [24304e97] FINISH, GetHardwareInfoVDSCommand, log id: 248c0dfc
>2018-05-29 10:25:02,105-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [532349c1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:25:02,113-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [4602ba3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:25:02,120-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [304533f0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:25:02,127-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [304533f0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 70dfe97d
>2018-05-29 10:25:02,133-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [43eb1040] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 274726b
>2018-05-29 10:25:02,135-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [43eb1040] FINISH, GetHardwareInfoVDSCommand, log id: 274726b
>2018-05-29 10:25:02,148-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [4014cf2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:25:02,153-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [413caaae] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:25:02,158-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [51ef3fb2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:25:02,164-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [51ef3fb2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f8d8d2
>2018-05-29 10:25:02,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [304533f0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:25:02,508-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [304533f0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 70dfe97d
>2018-05-29 10:25:02,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [304533f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:25:02,513-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [304533f0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:25:02,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [304533f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:25:02,518-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [304533f0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:25:02,522-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [304533f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:25:02,523-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [304533f0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:25:02,525-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [304533f0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:25:02,528-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [304533f0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 304533f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:25:02,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 75c01cef
>2018-05-29 10:25:02,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:25:02,635-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:25:02,635-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:25:02,635-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:25:02,635-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:25:02,635-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 75c01cef
>2018-05-29 10:25:02,635-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:25:02,648-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [51a9c447] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:25:02,649-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [51a9c447] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4f2c6494
>2018-05-29 10:25:02,652-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [51a9c447] FINISH, SetVdsStatusVDSCommand, log id: 4f2c6494
>2018-05-29 10:25:02,685-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [51a9c447] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 51a9c447, Job ID: e0b635ba-638b-4327-ae56-a13a051678b8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:25:02,690-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [51a9c447] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4602ba3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:25:02,692-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [51a9c447] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4602ba3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:25:02,705-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [387707f7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:25:02,707-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [387707f7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:25:03,537-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [51ef3fb2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:25:03,537-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [51ef3fb2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6f8d8d2
>2018-05-29 10:25:03,541-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [51ef3fb2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:25:03,542-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [51ef3fb2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:25:03,546-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [51ef3fb2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:25:03,547-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [51ef3fb2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:25:03,551-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [51ef3fb2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:25:03,552-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [51ef3fb2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:25:03,556-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [51ef3fb2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:25:03,559-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [51ef3fb2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 51ef3fb2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:25:03,562-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 17302ecd
>2018-05-29 10:25:03,562-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:25:03,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:25:03,659-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:25:03,659-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:25:03,659-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:25:03,659-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 17302ecd
>2018-05-29 10:25:03,659-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:25:03,673-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [69d87ee1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:25:03,674-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [69d87ee1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6528a2ce
>2018-05-29 10:25:03,677-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [69d87ee1] FINISH, SetVdsStatusVDSCommand, log id: 6528a2ce
>2018-05-29 10:25:03,682-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d87ee1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 69d87ee1, Job ID: e18cbe09-280f-4588-b790-97f065504cd4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:25:03,687-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d87ee1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 413caaae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:25:03,689-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d87ee1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 413caaae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:25:03,699-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [3857f01d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:25:03,701-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [3857f01d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3d07043f] Autorecovering 2 hosts
>2018-05-29 10:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3d07043f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:30:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2ee8f4dd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:30:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2ee8f4dd] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:30:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2ee8f4dd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:30:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2ee8f4dd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:30:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2ee8f4dd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3836b300
>2018-05-29 10:30:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2ee8f4dd] FINISH, SetVdsStatusVDSCommand, log id: 3836b300
>2018-05-29 10:30:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2ee8f4dd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:30:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2ee8f4dd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:30:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2ee8f4dd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:30:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7ed4ba38] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:30:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7ed4ba38] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:30:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7ed4ba38] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:30:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7ed4ba38] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:30:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7ed4ba38] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 55f66a56
>2018-05-29 10:30:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7ed4ba38] FINISH, SetVdsStatusVDSCommand, log id: 55f66a56
>2018-05-29 10:30:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [7ed4ba38] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7ca3fbfe
>2018-05-29 10:30:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [7ed4ba38] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7ca3fbfe
>2018-05-29 10:30:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7ed4ba38] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:30:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7ed4ba38] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:30:00,273-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6cfa4b20] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 593d230f
>2018-05-29 10:30:00,275-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6cfa4b20] FINISH, GetHardwareInfoVDSCommand, log id: 593d230f
>2018-05-29 10:30:00,288-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [380e1b13] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:30:00,296-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1d9798db] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:30:00,302-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2078d137] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:30:00,314-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2078d137] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7648b9fd
>2018-05-29 10:30:00,699-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2078d137] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:30:00,699-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2078d137] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7648b9fd
>2018-05-29 10:30:00,703-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2078d137] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:30:00,704-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2078d137] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:30:00,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2078d137] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:30:00,708-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2078d137] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:30:00,713-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2078d137] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:30:00,713-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2078d137] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:30:00,716-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2078d137] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:30:00,719-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2078d137] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2078d137, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:30:00,725-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4cae0e22
>2018-05-29 10:30:00,725-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:30:00,824-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:30:00,824-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:30:00,824-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:30:00,824-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:30:00,824-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4cae0e22
>2018-05-29 10:30:00,824-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:30:00,838-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [6a1addd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:30:00,840-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6a1addd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4381fbb2
>2018-05-29 10:30:00,842-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6a1addd] FINISH, SetVdsStatusVDSCommand, log id: 4381fbb2
>2018-05-29 10:30:00,873-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a1addd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6a1addd, Job ID: 78921673-736d-42c5-a943-84c3d196e8d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:30:00,878-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a1addd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d9798db, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:30:00,880-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a1addd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d9798db, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:30:00,891-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [50cf7693] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:30:00,892-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [50cf7693] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:30:01,463-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4e51b8b9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 750b5739
>2018-05-29 10:30:01,464-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4e51b8b9] FINISH, GetHardwareInfoVDSCommand, log id: 750b5739
>2018-05-29 10:30:01,477-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [790d6a9a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:30:01,482-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [1aebf501] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:30:01,487-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [485ad85e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:30:01,493-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [485ad85e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c260e26
>2018-05-29 10:30:01,861-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [485ad85e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:30:01,861-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [485ad85e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c260e26
>2018-05-29 10:30:01,865-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [485ad85e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:30:01,865-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [485ad85e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:30:01,869-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [485ad85e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:30:01,870-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [485ad85e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:30:01,874-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [485ad85e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:30:01,875-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [485ad85e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:30:01,878-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [485ad85e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:30:01,880-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [485ad85e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 485ad85e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:30:01,883-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 182174bd
>2018-05-29 10:30:01,883-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:30:01,980-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:30:01,980-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:30:01,980-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:30:01,980-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:30:01,980-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 182174bd
>2018-05-29 10:30:01,980-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:30:01,993-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [6fec5a73] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:30:01,995-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6fec5a73] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f99b5a6
>2018-05-29 10:30:01,997-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6fec5a73] FINISH, SetVdsStatusVDSCommand, log id: f99b5a6
>2018-05-29 10:30:02,002-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6fec5a73] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6fec5a73, Job ID: 0462cd05-70db-42d6-aee6-143c2b8b08f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:30:02,007-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6fec5a73] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1aebf501, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:30:02,009-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6fec5a73] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1aebf501, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:30:02,019-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [505af95b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:30:02,020-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [505af95b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3dd1a564] Autorecovering 2 hosts
>2018-05-29 10:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3dd1a564] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:35:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [604a98e3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:35:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [604a98e3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:35:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [604a98e3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:35:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [604a98e3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:35:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [604a98e3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6e16023d
>2018-05-29 10:35:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [604a98e3] FINISH, SetVdsStatusVDSCommand, log id: 6e16023d
>2018-05-29 10:35:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [604a98e3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:35:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [604a98e3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:35:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [604a98e3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:35:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6e915364] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:35:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6e915364] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:35:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6e915364] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:35:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6e915364] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:35:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6e915364] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 480abed1
>2018-05-29 10:35:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6e915364] FINISH, SetVdsStatusVDSCommand, log id: 480abed1
>2018-05-29 10:35:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6e915364] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 391498d5
>2018-05-29 10:35:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6e915364] FINISH, SetHaMaintenanceModeVDSCommand, log id: 391498d5
>2018-05-29 10:35:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6e915364] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:35:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6e915364] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:35:01,766-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [10cb05f8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 184f6863
>2018-05-29 10:35:01,768-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [10cb05f8] FINISH, GetHardwareInfoVDSCommand, log id: 184f6863
>2018-05-29 10:35:01,781-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [36a4de55] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:35:01,789-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [6ac7cd8e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:35:01,795-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [918de9c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:35:01,807-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [918de9c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6dfa72d9
>2018-05-29 10:35:02,196-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [918de9c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:35:02,196-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [918de9c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6dfa72d9
>2018-05-29 10:35:02,200-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [918de9c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:35:02,200-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [918de9c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:35:02,205-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [918de9c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:35:02,205-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [918de9c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:35:02,209-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [918de9c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:35:02,209-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [918de9c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:35:02,213-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [918de9c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:35:02,215-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [918de9c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 918de9c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:35:02,222-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1cd4b8bd
>2018-05-29 10:35:02,222-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:35:02,321-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:35:02,321-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:35:02,321-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:35:02,321-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:35:02,321-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1cd4b8bd
>2018-05-29 10:35:02,321-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:35:02,335-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [4878fc3b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:35:02,336-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4878fc3b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a17031f
>2018-05-29 10:35:02,338-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4878fc3b] FINISH, SetVdsStatusVDSCommand, log id: 5a17031f
>2018-05-29 10:35:02,372-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4878fc3b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4878fc3b, Job ID: 73554457-71c8-42c9-90af-6eef72801f6c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:35:02,376-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4878fc3b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6ac7cd8e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:35:02,379-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4878fc3b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6ac7cd8e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:35:02,390-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [a1ee432] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:35:02,391-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [a1ee432] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:35:02,815-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [57ff2501] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 43c49bb1
>2018-05-29 10:35:02,817-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [57ff2501] FINISH, GetHardwareInfoVDSCommand, log id: 43c49bb1
>2018-05-29 10:35:02,831-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [3bea6ddb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:35:02,835-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [73a3ce2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:35:02,841-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3c870efc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:35:02,848-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3c870efc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1ab81c4d
>2018-05-29 10:35:03,213-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c870efc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:35:03,213-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3c870efc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1ab81c4d
>2018-05-29 10:35:03,217-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c870efc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:35:03,218-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3c870efc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:35:03,222-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c870efc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:35:03,223-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3c870efc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:35:03,227-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c870efc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:35:03,227-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3c870efc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:35:03,230-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3c870efc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:35:03,233-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c870efc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3c870efc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:35:03,236-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7c65a65
>2018-05-29 10:35:03,236-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:35:03,333-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:35:03,333-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:35:03,333-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:35:03,333-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:35:03,333-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7c65a65
>2018-05-29 10:35:03,333-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:35:03,346-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [7457f5dc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:35:03,347-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7457f5dc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3594744d
>2018-05-29 10:35:03,350-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7457f5dc] FINISH, SetVdsStatusVDSCommand, log id: 3594744d
>2018-05-29 10:35:03,355-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7457f5dc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7457f5dc, Job ID: 400778d9-fc76-44fb-94b4-9f6ea8bd8044, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:35:03,360-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7457f5dc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 73a3ce2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:35:03,362-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7457f5dc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 73a3ce2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:35:03,372-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [59fa878a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:35:03,374-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [59fa878a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [599b8ac2] Autorecovering 2 hosts
>2018-05-29 10:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [599b8ac2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:40:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [556648dd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:40:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [556648dd] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:40:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [556648dd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:40:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [556648dd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:40:00,050-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [556648dd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 114aa52f
>2018-05-29 10:40:00,052-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [556648dd] FINISH, SetVdsStatusVDSCommand, log id: 114aa52f
>2018-05-29 10:40:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [556648dd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:40:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [556648dd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:40:00,056-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [556648dd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:40:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [ba8ea82] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:40:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [ba8ea82] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:40:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [ba8ea82] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:40:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [ba8ea82] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:40:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [ba8ea82] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 11d69afc
>2018-05-29 10:40:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [ba8ea82] FINISH, SetVdsStatusVDSCommand, log id: 11d69afc
>2018-05-29 10:40:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [ba8ea82] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 15269ec7
>2018-05-29 10:40:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [ba8ea82] FINISH, SetHaMaintenanceModeVDSCommand, log id: 15269ec7
>2018-05-29 10:40:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [ba8ea82] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:40:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [ba8ea82] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:40:01,045-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1924df55] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2bd4295b
>2018-05-29 10:40:01,047-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1924df55] FINISH, GetHardwareInfoVDSCommand, log id: 2bd4295b
>2018-05-29 10:40:01,060-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [47dc7b9a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:40:01,068-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [496c50ca] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:40:01,074-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [41f698ad] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:40:01,080-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [41f698ad] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 206d80d
>2018-05-29 10:40:01,445-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41f698ad] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:40:01,445-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [41f698ad] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 206d80d
>2018-05-29 10:40:01,449-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41f698ad] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:40:01,449-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41f698ad] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:40:01,453-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41f698ad] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:40:01,454-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41f698ad] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:40:01,458-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41f698ad] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:40:01,458-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41f698ad] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:40:01,461-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [41f698ad] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:40:01,463-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41f698ad] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 41f698ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:40:01,466-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5e672d90
>2018-05-29 10:40:01,466-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:40:01,562-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:40:01,562-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:40:01,562-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:40:01,562-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:40:01,562-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5e672d90
>2018-05-29 10:40:01,562-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:40:01,575-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [6675e563] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:40:01,576-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6675e563] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3af3907
>2018-05-29 10:40:01,579-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6675e563] FINISH, SetVdsStatusVDSCommand, log id: 3af3907
>2018-05-29 10:40:01,607-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6675e563] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6675e563, Job ID: 086eb36a-dd0b-44cd-bd50-b63b6f746065, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:40:01,612-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6675e563] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 496c50ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:40:01,614-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6675e563] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 496c50ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:40:01,625-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [12928d85] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:40:01,626-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [12928d85] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:40:03,078-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [6e0dffcc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 71d3f3d2
>2018-05-29 10:40:03,080-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [6e0dffcc] FINISH, GetHardwareInfoVDSCommand, log id: 71d3f3d2
>2018-05-29 10:40:03,094-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [13acaf69] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:40:03,098-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [271ec1dd] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:40:03,104-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2201c548] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:40:03,111-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2201c548] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 14d13051
>2018-05-29 10:40:03,489-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2201c548] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:40:03,489-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2201c548] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 14d13051
>2018-05-29 10:40:03,493-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2201c548] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:40:03,494-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2201c548] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:40:03,498-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2201c548] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:40:03,499-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2201c548] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:40:03,503-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2201c548] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:40:03,504-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2201c548] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:40:03,508-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2201c548] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:40:03,510-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2201c548] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2201c548, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:40:03,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 77d6cf58
>2018-05-29 10:40:03,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:40:03,610-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:40:03,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:40:03,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:40:03,610-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:40:03,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 77d6cf58
>2018-05-29 10:40:03,610-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:40:03,623-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [5bd223a8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:40:03,624-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5bd223a8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d4159f9
>2018-05-29 10:40:03,627-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5bd223a8] FINISH, SetVdsStatusVDSCommand, log id: d4159f9
>2018-05-29 10:40:03,632-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5bd223a8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5bd223a8, Job ID: 38da2e48-83b5-40e7-94e3-47a217d25728, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:40:03,637-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5bd223a8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 271ec1dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:40:03,639-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5bd223a8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 271ec1dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:40:03,649-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [3f7d3c86] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:40:03,650-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [3f7d3c86] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:45:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [13d95a29] Autorecovering 2 hosts
>2018-05-29 10:45:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [13d95a29] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:45:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4d162ca0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:45:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4d162ca0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:45:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4d162ca0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:45:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4d162ca0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:45:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4d162ca0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7fde0393
>2018-05-29 10:45:00,045-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4d162ca0] FINISH, SetVdsStatusVDSCommand, log id: 7fde0393
>2018-05-29 10:45:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4d162ca0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:45:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4d162ca0] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:45:00,047-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [4d162ca0] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:45:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60c6021a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:45:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60c6021a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:45:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60c6021a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:45:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60c6021a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:45:00,062-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [60c6021a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30f07d6e
>2018-05-29 10:45:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [60c6021a] FINISH, SetVdsStatusVDSCommand, log id: 30f07d6e
>2018-05-29 10:45:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [60c6021a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 69b187c1
>2018-05-29 10:45:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [60c6021a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 69b187c1
>2018-05-29 10:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60c6021a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60c6021a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:45:01,255-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5f4c97b8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 32476638
>2018-05-29 10:45:01,257-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5f4c97b8] FINISH, GetHardwareInfoVDSCommand, log id: 32476638
>2018-05-29 10:45:01,276-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [3ffacdad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:45:01,280-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [66ab26d8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:45:01,286-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [ac9129b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:45:01,344-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [ac9129b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4e9d973b
>2018-05-29 10:45:01,721-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [ac9129b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:45:01,721-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [ac9129b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4e9d973b
>2018-05-29 10:45:01,725-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [ac9129b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:45:01,726-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [ac9129b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:45:01,730-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [ac9129b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:45:01,731-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [ac9129b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:45:01,735-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [ac9129b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:45:01,736-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [ac9129b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:45:01,738-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [ac9129b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:45:01,741-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [ac9129b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: ac9129b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:45:01,748-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 253a5ed0
>2018-05-29 10:45:01,748-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:45:01,847-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:45:01,848-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:45:01,848-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:45:01,848-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:45:01,848-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 253a5ed0
>2018-05-29 10:45:01,848-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:45:01,862-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [16b223b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:45:01,863-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [16b223b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1e3fa775
>2018-05-29 10:45:01,865-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [16b223b] FINISH, SetVdsStatusVDSCommand, log id: 1e3fa775
>2018-05-29 10:45:01,873-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16b223b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 16b223b, Job ID: 61d4eece-ce81-4724-8a25-d445b2878d19, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:45:01,877-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16b223b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 66ab26d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:45:01,880-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16b223b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 66ab26d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:45:01,891-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [4bc4fb42] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:45:01,892-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [4bc4fb42] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:45:02,309-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2566c801] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5f4f8788
>2018-05-29 10:45:02,311-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2566c801] FINISH, GetHardwareInfoVDSCommand, log id: 5f4f8788
>2018-05-29 10:45:02,324-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [67cd41d7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:45:02,329-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5ca41c45] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:45:02,334-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3c04d2ec] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:45:02,341-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3c04d2ec] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 10305958
>2018-05-29 10:45:04,656-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c04d2ec] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:45:04,656-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3c04d2ec] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 10305958
>2018-05-29 10:45:04,660-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c04d2ec] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:45:04,661-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3c04d2ec] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:45:04,665-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c04d2ec] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:45:04,666-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3c04d2ec] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:45:04,670-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c04d2ec] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:45:04,670-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3c04d2ec] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:45:04,674-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3c04d2ec] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:45:04,676-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c04d2ec] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3c04d2ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:45:04,679-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 11ca579c
>2018-05-29 10:45:04,679-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:45:04,776-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:45:04,776-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:45:04,776-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:45:04,776-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:45:04,777-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 11ca579c
>2018-05-29 10:45:04,777-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:45:04,790-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [46381caf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:45:04,791-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [46381caf] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f6f0503
>2018-05-29 10:45:04,794-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [46381caf] FINISH, SetVdsStatusVDSCommand, log id: 5f6f0503
>2018-05-29 10:45:04,825-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [46381caf] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 46381caf, Job ID: 360d0409-3de8-48c9-84e9-83bfce8af333, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:45:04,830-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [46381caf] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5ca41c45, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:45:04,832-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [46381caf] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5ca41c45, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:45:04,843-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [306ff3e3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:45:04,844-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [306ff3e3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [17fefa0b] Autorecovering 2 hosts
>2018-05-29 10:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [17fefa0b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:50:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [74a41539] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [74a41539] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [74a41539] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [74a41539] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:50:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [74a41539] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 69ce828a
>2018-05-29 10:50:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [74a41539] FINISH, SetVdsStatusVDSCommand, log id: 69ce828a
>2018-05-29 10:50:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [74a41539] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:50:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [74a41539] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:50:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [74a41539] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:50:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [46f05fa7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:50:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [46f05fa7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:50:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [46f05fa7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:50:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [46f05fa7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:50:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [46f05fa7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2870e919
>2018-05-29 10:50:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [46f05fa7] FINISH, SetVdsStatusVDSCommand, log id: 2870e919
>2018-05-29 10:50:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [46f05fa7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6bec82a0
>2018-05-29 10:50:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [46f05fa7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6bec82a0
>2018-05-29 10:50:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [46f05fa7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:50:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [46f05fa7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:50:02,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [962f45d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6da673ae
>2018-05-29 10:50:02,524-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [962f45d] FINISH, GetHardwareInfoVDSCommand, log id: 6da673ae
>2018-05-29 10:50:02,538-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [5780c8c8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:50:02,544-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [30bbc828] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3f1266b2
>2018-05-29 10:50:02,545-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [30bbc828] FINISH, GetHardwareInfoVDSCommand, log id: 3f1266b2
>2018-05-29 10:50:02,547-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [58d710eb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:50:02,553-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [77a9fe92] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:50:02,560-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [465e1275] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:50:02,564-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [50cfbad1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:50:02,570-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [450d5cba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:50:02,583-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [450d5cba] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 666d6b50
>2018-05-29 10:50:02,613-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [77a9fe92] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 615f7765
>2018-05-29 10:50:02,977-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [450d5cba] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:50:02,977-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [450d5cba] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 666d6b50
>2018-05-29 10:50:02,981-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [450d5cba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:50:02,982-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [450d5cba] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:50:02,986-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [450d5cba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:50:02,987-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [450d5cba] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:50:02,989-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77a9fe92] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:50:02,989-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [77a9fe92] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 615f7765
>2018-05-29 10:50:02,991-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [450d5cba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:50:02,992-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [450d5cba] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:50:02,993-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77a9fe92] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:50:02,994-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [77a9fe92] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:50:02,995-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [450d5cba] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:50:02,997-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77a9fe92] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:50:02,998-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [77a9fe92] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:50:02,998-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [450d5cba] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 450d5cba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:50:03,002-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77a9fe92] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:50:03,002-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [77a9fe92] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:50:03,005-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [77a9fe92] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:50:03,005-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4e98fbdc
>2018-05-29 10:50:03,005-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:50:03,008-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77a9fe92] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 77a9fe92, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:50:03,104-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:50:03,104-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:50:03,104-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:50:03,105-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:50:03,105-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4e98fbdc
>2018-05-29 10:50:03,105-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:50:03,110-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5cd9da0b
>2018-05-29 10:50:03,110-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:50:03,118-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [2fb734d8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:50:03,119-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2fb734d8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19e7ec0a
>2018-05-29 10:50:03,122-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2fb734d8] FINISH, SetVdsStatusVDSCommand, log id: 19e7ec0a
>2018-05-29 10:50:03,155-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fb734d8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2fb734d8, Job ID: 57401dda-8e93-4830-b457-8caeab68c7b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:50:03,159-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fb734d8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 50cfbad1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:50:03,161-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fb734d8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 50cfbad1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:50:03,172-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [92d63b0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:50:03,173-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [92d63b0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:50:03,212-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:50:03,212-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:50:03,212-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:50:03,212-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:50:03,212-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5cd9da0b
>2018-05-29 10:50:03,212-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:50:03,225-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [674043ec] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:50:03,226-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [674043ec] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f8078a2
>2018-05-29 10:50:03,229-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [674043ec] FINISH, SetVdsStatusVDSCommand, log id: 1f8078a2
>2018-05-29 10:50:03,234-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [674043ec] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 674043ec, Job ID: 3f362d54-5601-46d6-a203-444b59967aed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:50:03,239-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [674043ec] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 58d710eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:50:03,241-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [674043ec] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 58d710eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:50:03,251-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [144da548] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:50:03,252-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [144da548] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:54:42,991-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler8) [336fb7b0] Attempting to update VMs/Templates Ovf.
>2018-05-29 10:54:42,994-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 10:54:42,994-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 10:54:42,994-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:54:42,999-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 10:54:43,002-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 10:54:43,002-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 10:54:43,002-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 10:54:43,002-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 10:54:43,003-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 10:54:43,003-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 10:55:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3be407c0] Autorecovering 2 hosts
>2018-05-29 10:55:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3be407c0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:55:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49e36883] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:55:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49e36883] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:55:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49e36883] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:55:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49e36883] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:55:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [49e36883] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7bb70d5d
>2018-05-29 10:55:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [49e36883] FINISH, SetVdsStatusVDSCommand, log id: 7bb70d5d
>2018-05-29 10:55:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49e36883] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:55:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49e36883] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 10:55:00,068-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [49e36883] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:55:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56e740ce] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:55:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56e740ce] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 10:55:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56e740ce] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:55:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56e740ce] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:55:00,082-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [56e740ce] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6fd894ea
>2018-05-29 10:55:00,084-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [56e740ce] FINISH, SetVdsStatusVDSCommand, log id: 6fd894ea
>2018-05-29 10:55:00,086-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [56e740ce] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 644aa55a
>2018-05-29 10:55:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [56e740ce] FINISH, SetHaMaintenanceModeVDSCommand, log id: 644aa55a
>2018-05-29 10:55:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56e740ce] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 10:55:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56e740ce] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 10:55:00,828-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [21d28557] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5c33b7b1
>2018-05-29 10:55:00,830-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [21d28557] FINISH, GetHardwareInfoVDSCommand, log id: 5c33b7b1
>2018-05-29 10:55:00,841-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [29040ba7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5f32e66c
>2018-05-29 10:55:00,843-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [29040ba7] FINISH, GetHardwareInfoVDSCommand, log id: 5f32e66c
>2018-05-29 10:55:00,846-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [42c87c56] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:55:00,855-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [a8b467a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:55:00,856-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [46977de7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:55:00,861-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5ffc0509] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:55:00,865-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [584950e5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:55:00,870-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [5b9e570] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 10:55:00,873-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5ffc0509] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 64368a8
>2018-05-29 10:55:00,877-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [5b9e570] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 39df2bc6
>2018-05-29 10:55:01,242-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b9e570] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:55:01,242-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [5b9e570] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 39df2bc6
>2018-05-29 10:55:01,246-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b9e570] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:55:01,247-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5b9e570] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:55:01,251-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5ffc0509] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 10:55:01,251-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5ffc0509] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 64368a8
>2018-05-29 10:55:01,252-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b9e570] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:55:01,253-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5b9e570] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:55:01,255-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5ffc0509] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:55:01,256-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5ffc0509] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:55:01,259-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b9e570] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:55:01,260-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5b9e570] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:55:01,260-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5ffc0509] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:55:01,261-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5ffc0509] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:55:01,263-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [5b9e570] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:55:01,265-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5ffc0509] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 10:55:01,266-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5ffc0509] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 10:55:01,266-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b9e570] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5b9e570, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:55:01,269-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5ffc0509] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 10:55:01,269-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 73813524
>2018-05-29 10:55:01,269-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:55:01,272-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5ffc0509] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5ffc0509, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 10:55:01,370-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:55:01,370-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:55:01,370-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:55:01,370-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:55:01,370-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 73813524
>2018-05-29 10:55:01,370-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:55:01,376-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 9eb20d4
>2018-05-29 10:55:01,376-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 10:55:01,383-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [43eb8575] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:55:01,384-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [43eb8575] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 490a9983
>2018-05-29 10:55:01,386-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [43eb8575] FINISH, SetVdsStatusVDSCommand, log id: 490a9983
>2018-05-29 10:55:01,417-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [43eb8575] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 43eb8575, Job ID: fc267b7e-a343-4ab3-b474-07e9b8540480, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:55:01,422-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [43eb8575] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 584950e5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:55:01,424-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [43eb8575] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 584950e5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:55:01,434-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [347879c2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 10:55:01,435-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [347879c2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 10:55:01,476-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:55:01,476-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 10:55:01,476-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 10:55:01,477-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 10:55:01,477-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 9eb20d4
>2018-05-29 10:55:01,477-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 10:55:01,489-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2201dff8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:55:01,491-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2201dff8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42246adb
>2018-05-29 10:55:01,492-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2201dff8] FINISH, SetVdsStatusVDSCommand, log id: 42246adb
>2018-05-29 10:55:01,498-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2201dff8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2201dff8, Job ID: 43dc1931-9d2e-4a32-b6b5-59f0a6ce6469, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 10:55:01,502-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2201dff8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: a8b467a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 10:55:01,504-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2201dff8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: a8b467a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 10:55:01,515-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [4d639180] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 10:55:01,516-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [4d639180] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:00:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [4c8f1ba6] Autorecovering 2 hosts
>2018-05-29 11:00:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [4c8f1ba6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:00:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6685af5c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:00:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6685af5c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:00:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6685af5c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:00:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6685af5c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:00:00,043-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6685af5c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ce68d51
>2018-05-29 11:00:00,046-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6685af5c] FINISH, SetVdsStatusVDSCommand, log id: ce68d51
>2018-05-29 11:00:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6685af5c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:00:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6685af5c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:00:00,048-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [6685af5c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:00:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60042523] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:00:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60042523] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:00:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60042523] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:00:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60042523] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:00:00,062-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [60042523] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 23007c79
>2018-05-29 11:00:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [60042523] FINISH, SetVdsStatusVDSCommand, log id: 23007c79
>2018-05-29 11:00:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [60042523] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 77c143a6
>2018-05-29 11:00:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [60042523] FINISH, SetHaMaintenanceModeVDSCommand, log id: 77c143a6
>2018-05-29 11:00:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60042523] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:00:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60042523] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:00:02,220-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3bbed291] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 172de13f
>2018-05-29 11:00:02,221-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [38701ba9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 589a03df
>2018-05-29 11:00:02,222-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3bbed291] FINISH, GetHardwareInfoVDSCommand, log id: 172de13f
>2018-05-29 11:00:02,223-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [38701ba9] FINISH, GetHardwareInfoVDSCommand, log id: 589a03df
>2018-05-29 11:00:02,238-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [3e3129e3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:00:02,239-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [52cd103a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:00:02,245-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5737561] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:00:02,248-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [65e4329] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:00:02,250-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7552f955] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:00:02,254-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3d431ade] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:00:02,263-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7552f955] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 21e71fc8
>2018-05-29 11:00:02,311-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3d431ade] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3391cee0
>2018-05-29 11:00:02,644-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7552f955] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:00:02,644-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7552f955] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 21e71fc8
>2018-05-29 11:00:02,648-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7552f955] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:00:02,649-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7552f955] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:00:02,653-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7552f955] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:00:02,653-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7552f955] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:00:02,657-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7552f955] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:00:02,658-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7552f955] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:00:02,661-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7552f955] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:00:02,663-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7552f955] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7552f955, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:00:02,670-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: e6b5b45
>2018-05-29 11:00:02,670-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:00:02,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3d431ade] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:00:02,680-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3d431ade] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3391cee0
>2018-05-29 11:00:02,684-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3d431ade] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:00:02,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3d431ade] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:00:02,688-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3d431ade] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:00:02,689-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3d431ade] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:00:02,693-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3d431ade] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:00:02,694-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3d431ade] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:00:02,696-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3d431ade] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:00:02,699-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3d431ade] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3d431ade, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:00:02,771-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:00:02,771-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:00:02,771-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:00:02,771-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:00:02,771-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: e6b5b45
>2018-05-29 11:00:02,771-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:00:02,773-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: c5137ec
>2018-05-29 11:00:02,773-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:00:02,785-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [7ddf80ca] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:00:02,786-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7ddf80ca] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 700f9e6d
>2018-05-29 11:00:02,788-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7ddf80ca] FINISH, SetVdsStatusVDSCommand, log id: 700f9e6d
>2018-05-29 11:00:02,796-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ddf80ca] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7ddf80ca, Job ID: 2d4e9cdc-80a0-4b89-9fea-c1ed1d3ee315, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:00:02,801-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ddf80ca] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5737561, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:00:02,804-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ddf80ca] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5737561, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:00:02,814-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [1699c707] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:00:02,815-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [1699c707] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:00:02,869-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:00:02,870-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:00:02,870-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:00:02,870-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:00:02,870-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: c5137ec
>2018-05-29 11:00:02,870-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:00:02,883-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [22fd26c2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:00:02,884-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [22fd26c2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 48ea90ad
>2018-05-29 11:00:02,886-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [22fd26c2] FINISH, SetVdsStatusVDSCommand, log id: 48ea90ad
>2018-05-29 11:00:02,892-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [22fd26c2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22fd26c2, Job ID: 93d3c489-4dfa-4b65-b21a-3e06e9051fe2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:00:02,896-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [22fd26c2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 65e4329, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:00:02,898-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [22fd26c2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 65e4329, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:00:02,908-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [9cc2aa4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:00:02,909-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [9cc2aa4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:05:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1d2d1677] Autorecovering 2 hosts
>2018-05-29 11:05:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1d2d1677] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:05:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [21c110b2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:05:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [21c110b2] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:05:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [21c110b2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:05:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [21c110b2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:05:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [21c110b2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7b126345
>2018-05-29 11:05:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [21c110b2] FINISH, SetVdsStatusVDSCommand, log id: 7b126345
>2018-05-29 11:05:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [21c110b2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:05:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [21c110b2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:05:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [21c110b2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:05:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [79499da] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:05:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [79499da] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:05:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [79499da] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:05:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [79499da] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:05:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [79499da] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1e582cfa
>2018-05-29 11:05:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [79499da] FINISH, SetVdsStatusVDSCommand, log id: 1e582cfa
>2018-05-29 11:05:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [79499da] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6888db54
>2018-05-29 11:05:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [79499da] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6888db54
>2018-05-29 11:05:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [79499da] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:05:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [79499da] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:05:00,445-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [16db8535] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 14592e8e
>2018-05-29 11:05:00,447-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [16db8535] FINISH, GetHardwareInfoVDSCommand, log id: 14592e8e
>2018-05-29 11:05:00,460-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7c98ad19] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:05:00,468-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [43d22ff7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:05:00,473-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [65ad710d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:05:00,484-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [65ad710d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6c84cba0
>2018-05-29 11:05:00,495-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [a74b77f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7c19c363
>2018-05-29 11:05:00,497-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [a74b77f] FINISH, GetHardwareInfoVDSCommand, log id: 7c19c363
>2018-05-29 11:05:00,509-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [60e82dea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:05:00,513-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3d9ed7a1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:05:00,519-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51dad4f3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:05:00,525-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51dad4f3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 542ff4c8
>2018-05-29 11:05:00,863-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [65ad710d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:05:00,863-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [65ad710d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6c84cba0
>2018-05-29 11:05:00,867-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [65ad710d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:05:00,868-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [65ad710d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:05:00,872-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [65ad710d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:05:00,872-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [65ad710d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:05:00,876-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [65ad710d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:05:00,877-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [65ad710d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:05:00,879-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [65ad710d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:05:00,882-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [65ad710d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 65ad710d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:05:00,889-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 765af215
>2018-05-29 11:05:00,889-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:05:00,899-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dad4f3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:05:00,899-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51dad4f3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 542ff4c8
>2018-05-29 11:05:00,903-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dad4f3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:05:00,904-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51dad4f3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:05:00,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dad4f3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:05:00,908-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51dad4f3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:05:00,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dad4f3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:05:00,912-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51dad4f3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:05:00,915-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51dad4f3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:05:00,918-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dad4f3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 51dad4f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:05:00,991-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:05:00,991-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:05:00,991-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:05:00,991-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:05:00,991-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 765af215
>2018-05-29 11:05:00,991-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:05:00,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 467a547c
>2018-05-29 11:05:00,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:05:01,004-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [2034a7ba] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:05:01,005-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2034a7ba] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f98ad3f
>2018-05-29 11:05:01,008-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2034a7ba] FINISH, SetVdsStatusVDSCommand, log id: 1f98ad3f
>2018-05-29 11:05:01,040-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2034a7ba] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2034a7ba, Job ID: ae7a8a4a-f55c-4152-9025-9913a03eb596, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:05:01,045-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2034a7ba] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 43d22ff7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:05:01,047-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2034a7ba] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 43d22ff7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:05:01,058-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [385609c1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:05:01,059-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [385609c1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:05:01,096-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:05:01,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:05:01,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:05:01,096-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:05:01,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 467a547c
>2018-05-29 11:05:01,096-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:05:01,109-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [336cd419] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:05:01,110-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [336cd419] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 129fa88d
>2018-05-29 11:05:01,112-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [336cd419] FINISH, SetVdsStatusVDSCommand, log id: 129fa88d
>2018-05-29 11:05:01,117-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [336cd419] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 336cd419, Job ID: b7e7dded-68a4-4382-a5ff-646127b22848, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:05:01,122-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [336cd419] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3d9ed7a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:05:01,124-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [336cd419] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3d9ed7a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:05:01,134-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [12559737] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:05:01,135-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [12559737] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [45fc4b29] Autorecovering 2 hosts
>2018-05-29 11:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [45fc4b29] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:10:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [e6baaf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:10:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [e6baaf] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:10:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [e6baaf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:10:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [e6baaf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:10:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [e6baaf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13d05da7
>2018-05-29 11:10:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [e6baaf] FINISH, SetVdsStatusVDSCommand, log id: 13d05da7
>2018-05-29 11:10:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [e6baaf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:10:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [e6baaf] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:10:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [e6baaf] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:10:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [caef888] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:10:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [caef888] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:10:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [caef888] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:10:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [caef888] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:10:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [caef888] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: dfc1893
>2018-05-29 11:10:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [caef888] FINISH, SetVdsStatusVDSCommand, log id: dfc1893
>2018-05-29 11:10:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [caef888] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 41a208fd
>2018-05-29 11:10:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [caef888] FINISH, SetHaMaintenanceModeVDSCommand, log id: 41a208fd
>2018-05-29 11:10:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [caef888] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:10:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [caef888] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:10:01,696-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3be8d7a7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: fe7d14e
>2018-05-29 11:10:01,698-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3be8d7a7] FINISH, GetHardwareInfoVDSCommand, log id: fe7d14e
>2018-05-29 11:10:01,711-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [2b0b1b0a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:10:01,719-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [66b2e8d4] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:10:01,725-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5a4014a8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:10:01,737-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5a4014a8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 98f63c5
>2018-05-29 11:10:01,747-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1507b31d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: dd0bf12
>2018-05-29 11:10:01,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1507b31d] FINISH, GetHardwareInfoVDSCommand, log id: dd0bf12
>2018-05-29 11:10:01,775-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [d988a95] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:10:01,783-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [2de64aa3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:10:01,788-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6efe4fc8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:10:01,847-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6efe4fc8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: f788aa1
>2018-05-29 11:10:02,219-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6efe4fc8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:10:02,220-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6efe4fc8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: f788aa1
>2018-05-29 11:10:02,223-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6efe4fc8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:10:02,224-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6efe4fc8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:10:02,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6efe4fc8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:10:02,229-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6efe4fc8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:10:02,233-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6efe4fc8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:10:02,233-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6efe4fc8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:10:02,236-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6efe4fc8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:10:02,239-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6efe4fc8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6efe4fc8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:10:02,245-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1411959a
>2018-05-29 11:10:02,245-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:10:02,343-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:10:02,343-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:10:02,343-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:10:02,343-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:10:02,343-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1411959a
>2018-05-29 11:10:02,343-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:10:02,357-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [1205f3a7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:10:02,358-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1205f3a7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50ab1644
>2018-05-29 11:10:02,361-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1205f3a7] FINISH, SetVdsStatusVDSCommand, log id: 50ab1644
>2018-05-29 11:10:02,393-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1205f3a7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1205f3a7, Job ID: c079ef67-a886-43cc-9ecb-9dd4c745db84, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:10:02,398-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1205f3a7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2de64aa3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:10:02,400-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1205f3a7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2de64aa3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:10:02,411-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [2fe2f488] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:10:02,412-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [2fe2f488] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:10:03,057-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a4014a8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:10:03,057-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5a4014a8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 98f63c5
>2018-05-29 11:10:03,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a4014a8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:10:03,062-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5a4014a8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:10:03,066-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a4014a8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:10:03,067-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5a4014a8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:10:03,071-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a4014a8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:10:03,071-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5a4014a8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:10:03,074-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5a4014a8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:10:03,077-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a4014a8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5a4014a8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:10:03,080-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 353760d6
>2018-05-29 11:10:03,080-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:10:03,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:10:03,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:10:03,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:10:03,178-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:10:03,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 353760d6
>2018-05-29 11:10:03,178-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:10:03,190-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [6801c788] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:10:03,192-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6801c788] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4d1c2b0e
>2018-05-29 11:10:03,194-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6801c788] FINISH, SetVdsStatusVDSCommand, log id: 4d1c2b0e
>2018-05-29 11:10:03,199-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6801c788] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6801c788, Job ID: 2c28e3aa-2ae2-4903-9ad5-7f13bf6a3c03, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:10:03,204-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6801c788] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 66b2e8d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:10:03,206-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6801c788] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 66b2e8d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:10:03,217-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [1467b4ec] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:10:03,218-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [1467b4ec] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [56f2f85f] Autorecovering 2 hosts
>2018-05-29 11:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [56f2f85f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:15:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [521db0f2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [521db0f2] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [521db0f2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [521db0f2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:15:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [521db0f2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76770731
>2018-05-29 11:15:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [521db0f2] FINISH, SetVdsStatusVDSCommand, log id: 76770731
>2018-05-29 11:15:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [521db0f2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:15:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [521db0f2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:15:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [521db0f2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:15:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [273135e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:15:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [273135e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:15:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [273135e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:15:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [273135e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:15:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [273135e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4350aca6
>2018-05-29 11:15:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [273135e] FINISH, SetVdsStatusVDSCommand, log id: 4350aca6
>2018-05-29 11:15:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [273135e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 32b46672
>2018-05-29 11:15:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [273135e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 32b46672
>2018-05-29 11:15:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [273135e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:15:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [273135e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:15:00,788-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3e14ed94] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 11117c82
>2018-05-29 11:15:00,789-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3e14ed94] FINISH, GetHardwareInfoVDSCommand, log id: 11117c82
>2018-05-29 11:15:00,802-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [54a0a9c9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:15:00,811-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [7e94d017] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:15:00,816-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [30eead02] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:15:00,829-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [30eead02] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2a5d9569
>2018-05-29 11:15:01,214-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [30eead02] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:15:01,214-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [30eead02] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2a5d9569
>2018-05-29 11:15:01,219-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [30eead02] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:15:01,219-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [30eead02] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:15:01,223-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [30eead02] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:15:01,224-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [30eead02] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:15:01,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [30eead02] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:15:01,228-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [30eead02] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:15:01,231-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [30eead02] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:15:01,234-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [30eead02] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 30eead02, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:15:01,240-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1971277c
>2018-05-29 11:15:01,240-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:15:01,339-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:15:01,339-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:15:01,339-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:15:01,339-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:15:01,339-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1971277c
>2018-05-29 11:15:01,339-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:15:01,352-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [482daa08] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:15:01,354-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [482daa08] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: c53acb9
>2018-05-29 11:15:01,356-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [482daa08] FINISH, SetVdsStatusVDSCommand, log id: c53acb9
>2018-05-29 11:15:01,387-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [482daa08] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 482daa08, Job ID: 3d14bd48-df2f-4a6a-8114-0e7baee8f39b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:15:01,391-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [482daa08] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7e94d017, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:15:01,394-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [482daa08] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7e94d017, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:15:01,405-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [50ca1bdd] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:15:01,406-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [50ca1bdd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:15:03,107-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [ec3d7ed] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 645b57
>2018-05-29 11:15:03,109-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [ec3d7ed] FINISH, GetHardwareInfoVDSCommand, log id: 645b57
>2018-05-29 11:15:03,122-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [7922bdc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:15:03,127-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [57c980b1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:15:03,132-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [46b3347f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:15:03,139-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [46b3347f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4a1106ac
>2018-05-29 11:15:03,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46b3347f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:15:03,508-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [46b3347f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4a1106ac
>2018-05-29 11:15:03,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46b3347f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:15:03,512-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [46b3347f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:15:03,516-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46b3347f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:15:03,516-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [46b3347f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:15:03,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46b3347f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:15:03,521-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [46b3347f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:15:03,523-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [46b3347f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:15:03,526-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46b3347f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 46b3347f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:15:03,529-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5b2e5205
>2018-05-29 11:15:03,529-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:15:03,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:15:03,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:15:03,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:15:03,627-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:15:03,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5b2e5205
>2018-05-29 11:15:03,627-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:15:03,639-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [3b4c998d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:15:03,640-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3b4c998d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3eca43b3
>2018-05-29 11:15:03,642-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3b4c998d] FINISH, SetVdsStatusVDSCommand, log id: 3eca43b3
>2018-05-29 11:15:03,648-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3b4c998d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3b4c998d, Job ID: 99ac2271-7d46-4f4e-a01e-69a285d791a7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:15:03,652-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3b4c998d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 57c980b1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:15:03,654-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3b4c998d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 57c980b1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:15:03,665-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2705ac49] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:15:03,666-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2705ac49] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:20:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4443b56d] Autorecovering 2 hosts
>2018-05-29 11:20:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4443b56d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:20:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2d3d0e8b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:20:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2d3d0e8b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:20:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2d3d0e8b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:20:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2d3d0e8b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:20:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2d3d0e8b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 40db560a
>2018-05-29 11:20:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2d3d0e8b] FINISH, SetVdsStatusVDSCommand, log id: 40db560a
>2018-05-29 11:20:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2d3d0e8b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:20:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2d3d0e8b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:20:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [2d3d0e8b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:20:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7904973c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:20:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7904973c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:20:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7904973c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:20:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7904973c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:20:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7904973c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1530d2ac
>2018-05-29 11:20:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7904973c] FINISH, SetVdsStatusVDSCommand, log id: 1530d2ac
>2018-05-29 11:20:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [7904973c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 56b3f4a9
>2018-05-29 11:20:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [7904973c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 56b3f4a9
>2018-05-29 11:20:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7904973c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:20:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7904973c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:20:01,338-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [433dbc7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6f4f5336
>2018-05-29 11:20:01,340-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [433dbc7] FINISH, GetHardwareInfoVDSCommand, log id: 6f4f5336
>2018-05-29 11:20:01,352-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [13bddd27] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:20:01,361-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [5bb12a08] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:20:01,366-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [35537a1d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:20:01,377-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [35537a1d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3d4369b4
>2018-05-29 11:20:02,268-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4b6b97c5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: d736fa5
>2018-05-29 11:20:02,269-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4b6b97c5] FINISH, GetHardwareInfoVDSCommand, log id: d736fa5
>2018-05-29 11:20:02,281-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [113a3a33] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:20:02,286-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [2c3018b4] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:20:02,291-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1a4936e9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:20:02,297-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1a4936e9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22ed814e
>2018-05-29 11:20:02,677-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1a4936e9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:20:02,677-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1a4936e9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 22ed814e
>2018-05-29 11:20:02,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1a4936e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:20:02,681-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1a4936e9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:20:02,685-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1a4936e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:20:02,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1a4936e9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:20:02,689-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1a4936e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:20:02,689-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1a4936e9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:20:02,692-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1a4936e9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:20:02,695-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1a4936e9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1a4936e9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:20:02,701-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2f8f0d62
>2018-05-29 11:20:02,701-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:20:02,748-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [35537a1d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:20:02,748-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [35537a1d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3d4369b4
>2018-05-29 11:20:02,752-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [35537a1d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:20:02,753-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [35537a1d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:20:02,756-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [35537a1d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:20:02,757-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [35537a1d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:20:02,761-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [35537a1d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:20:02,761-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [35537a1d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:20:02,764-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [35537a1d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:20:02,766-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [35537a1d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 35537a1d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:20:02,802-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:20:02,802-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:20:02,802-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:20:02,802-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:20:02,802-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2f8f0d62
>2018-05-29 11:20:02,802-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:20:02,803-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3e91d23f
>2018-05-29 11:20:02,803-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:20:02,815-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [699a388a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:20:02,816-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [699a388a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 9c05066
>2018-05-29 11:20:02,818-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [699a388a] FINISH, SetVdsStatusVDSCommand, log id: 9c05066
>2018-05-29 11:20:02,850-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [699a388a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 699a388a, Job ID: 909c7dd6-2355-49bb-88de-5c8377602227, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:20:02,855-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [699a388a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2c3018b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:20:02,857-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [699a388a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2c3018b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:20:02,867-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [64637d10] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:20:02,868-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [64637d10] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:20:02,903-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:20:02,903-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:20:02,903-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:20:02,903-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:20:02,903-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3e91d23f
>2018-05-29 11:20:02,903-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:20:02,915-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [53d7f324] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:20:02,916-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [53d7f324] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1aeb76ff
>2018-05-29 11:20:02,918-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [53d7f324] FINISH, SetVdsStatusVDSCommand, log id: 1aeb76ff
>2018-05-29 11:20:02,924-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [53d7f324] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 53d7f324, Job ID: 64637357-da7d-4802-a1c8-9dc68286e281, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:20:02,928-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [53d7f324] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5bb12a08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:20:02,930-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [53d7f324] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5bb12a08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:20:02,940-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [77aec4aa] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:20:02,941-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [77aec4aa] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:25:00,014-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [55f1e2e] Autorecovering 2 hosts
>2018-05-29 11:25:00,014-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [55f1e2e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:25:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [176b8a3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:25:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [176b8a3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:25:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [176b8a3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:25:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [176b8a3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:25:00,049-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [176b8a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5cc68a82
>2018-05-29 11:25:00,051-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [176b8a3] FINISH, SetVdsStatusVDSCommand, log id: 5cc68a82
>2018-05-29 11:25:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [176b8a3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:25:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [176b8a3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:25:00,055-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [176b8a3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:25:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b3a9f6e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:25:00,118-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b3a9f6e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:25:00,118-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b3a9f6e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:25:00,118-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b3a9f6e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:25:00,120-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5b3a9f6e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a4109dc
>2018-05-29 11:25:00,123-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5b3a9f6e] FINISH, SetVdsStatusVDSCommand, log id: a4109dc
>2018-05-29 11:25:00,126-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [5b3a9f6e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1196fd83
>2018-05-29 11:25:00,128-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [5b3a9f6e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1196fd83
>2018-05-29 11:25:00,128-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b3a9f6e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:25:00,130-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b3a9f6e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:25:00,530-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [72aa3282] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3ee134af
>2018-05-29 11:25:00,532-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [72aa3282] FINISH, GetHardwareInfoVDSCommand, log id: 3ee134af
>2018-05-29 11:25:00,545-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4059ff2a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:25:00,554-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [34c21a44] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:25:00,560-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6c5b4d4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:25:00,571-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6c5b4d4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5c7acad4
>2018-05-29 11:25:00,589-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [576a558d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3105e80b
>2018-05-29 11:25:00,591-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [576a558d] FINISH, GetHardwareInfoVDSCommand, log id: 3105e80b
>2018-05-29 11:25:00,604-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [d15ce10] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:25:00,608-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [786ed5d8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:25:00,614-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [52ca7968] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:25:00,620-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [52ca7968] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5056fda
>2018-05-29 11:25:00,949-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c5b4d4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:25:00,949-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6c5b4d4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5c7acad4
>2018-05-29 11:25:00,953-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c5b4d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:25:00,954-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6c5b4d4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:25:00,958-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c5b4d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:25:00,959-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6c5b4d4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:25:00,962-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c5b4d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:25:00,963-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6c5b4d4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:25:00,965-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6c5b4d4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:25:00,968-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c5b4d4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6c5b4d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:25:00,975-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 95c53d
>2018-05-29 11:25:00,975-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:25:00,986-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52ca7968] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:25:00,986-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [52ca7968] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5056fda
>2018-05-29 11:25:00,990-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52ca7968] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:25:00,991-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [52ca7968] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:25:00,994-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52ca7968] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:25:00,995-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [52ca7968] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:25:00,999-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52ca7968] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:25:00,999-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [52ca7968] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:25:01,002-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [52ca7968] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:25:01,004-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52ca7968] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 52ca7968, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:25:01,072-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:25:01,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:25:01,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:25:01,072-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:25:01,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 95c53d
>2018-05-29 11:25:01,072-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:25:01,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7dd1ee49
>2018-05-29 11:25:01,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:25:01,084-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [60b008ff] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:25:01,085-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [60b008ff] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d482290
>2018-05-29 11:25:01,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [60b008ff] FINISH, SetVdsStatusVDSCommand, log id: 3d482290
>2018-05-29 11:25:01,095-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [60b008ff] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 60b008ff, Job ID: 9c51a2e1-dfe2-4f96-b0a4-f15589823550, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:25:01,099-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [60b008ff] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 34c21a44, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:25:01,101-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [60b008ff] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 34c21a44, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:25:01,112-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [539f83f5] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:25:01,113-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [539f83f5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:25:01,172-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:25:01,172-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:25:01,172-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:25:01,172-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:25:01,172-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7dd1ee49
>2018-05-29 11:25:01,172-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:25:01,184-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5cc5e848] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:25:01,185-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5cc5e848] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1130838f
>2018-05-29 11:25:01,187-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5cc5e848] FINISH, SetVdsStatusVDSCommand, log id: 1130838f
>2018-05-29 11:25:01,192-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5cc5e848] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5cc5e848, Job ID: cf187848-4438-4a21-afdb-7ddc681c2ff3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:25:01,197-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5cc5e848] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 786ed5d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:25:01,199-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5cc5e848] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 786ed5d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:25:01,209-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [5d404089] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:25:01,210-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [5d404089] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7694baf8] Autorecovering 2 hosts
>2018-05-29 11:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7694baf8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:30:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [29e81195] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:30:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [29e81195] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:30:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [29e81195] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:30:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [29e81195] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:30:00,048-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [29e81195] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: fec497c
>2018-05-29 11:30:00,050-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [29e81195] FINISH, SetVdsStatusVDSCommand, log id: fec497c
>2018-05-29 11:30:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [29e81195] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:30:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [29e81195] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:30:00,053-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [29e81195] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:30:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4a21ace0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:30:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4a21ace0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:30:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4a21ace0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:30:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4a21ace0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:30:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4a21ace0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 67462207
>2018-05-29 11:30:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4a21ace0] FINISH, SetVdsStatusVDSCommand, log id: 67462207
>2018-05-29 11:30:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4a21ace0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: ce6e6e7
>2018-05-29 11:30:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4a21ace0] FINISH, SetHaMaintenanceModeVDSCommand, log id: ce6e6e7
>2018-05-29 11:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4a21ace0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:30:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4a21ace0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:30:01,767-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [209f2bbe] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 28f4b98a
>2018-05-29 11:30:01,769-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [209f2bbe] FINISH, GetHardwareInfoVDSCommand, log id: 28f4b98a
>2018-05-29 11:30:01,782-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [6dacfab2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:30:01,790-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [55a49ad3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:30:01,795-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [544d0d3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1ddeb1b4
>2018-05-29 11:30:01,796-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [543c9566] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:30:01,797-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [544d0d3] FINISH, GetHardwareInfoVDSCommand, log id: 1ddeb1b4
>2018-05-29 11:30:01,808-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [543c9566] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4f6b1c9d
>2018-05-29 11:30:01,810-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [28dfa796] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:30:01,818-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [ca8a8cb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:30:01,823-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [36dd7a9d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:30:01,836-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [36dd7a9d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2bd231e5
>2018-05-29 11:30:02,189-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [543c9566] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:30:02,189-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [543c9566] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4f6b1c9d
>2018-05-29 11:30:02,193-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [543c9566] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:30:02,193-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [543c9566] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:30:02,197-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [543c9566] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:30:02,198-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [543c9566] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:30:02,201-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [543c9566] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:30:02,202-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [543c9566] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:30:02,205-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [543c9566] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:30:02,208-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [543c9566] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 543c9566, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:30:02,214-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3fb8c36
>2018-05-29 11:30:02,214-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:30:02,314-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:30:02,314-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:30:02,314-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:30:02,314-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:30:02,314-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3fb8c36
>2018-05-29 11:30:02,314-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:30:02,350-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [39c5813f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:30:02,352-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [39c5813f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 509b215c
>2018-05-29 11:30:02,354-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [39c5813f] FINISH, SetVdsStatusVDSCommand, log id: 509b215c
>2018-05-29 11:30:02,385-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [39c5813f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 39c5813f, Job ID: e8528859-299c-4d68-b06d-22b9e63f2625, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:30:02,390-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [39c5813f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 55a49ad3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:30:02,392-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [39c5813f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 55a49ad3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:30:02,403-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [796663e5] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:30:02,404-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [796663e5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:30:04,152-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [36dd7a9d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:30:04,153-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [36dd7a9d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2bd231e5
>2018-05-29 11:30:04,156-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [36dd7a9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:30:04,157-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [36dd7a9d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:30:04,161-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [36dd7a9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:30:04,161-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [36dd7a9d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:30:04,165-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [36dd7a9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:30:04,166-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [36dd7a9d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:30:04,169-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [36dd7a9d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:30:04,171-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [36dd7a9d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 36dd7a9d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:30:04,174-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1c3bd2dc
>2018-05-29 11:30:04,174-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:30:04,272-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:30:04,272-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:30:04,272-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:30:04,272-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:30:04,272-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1c3bd2dc
>2018-05-29 11:30:04,273-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:30:04,285-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [4c0913c0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:30:04,286-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4c0913c0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 64df96c8
>2018-05-29 11:30:04,288-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4c0913c0] FINISH, SetVdsStatusVDSCommand, log id: 64df96c8
>2018-05-29 11:30:04,293-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c0913c0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4c0913c0, Job ID: ffc904fa-e69a-42fe-97ab-a44d3b94a445, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:30:04,298-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c0913c0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: ca8a8cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:30:04,300-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c0913c0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: ca8a8cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:30:04,310-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [4f03beaf] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:30:04,311-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [4f03beaf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:35:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [559f9970] Autorecovering 2 hosts
>2018-05-29 11:35:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [559f9970] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:35:00,011-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2a10c147] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:35:00,017-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2a10c147] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:35:00,017-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2a10c147] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:35:00,017-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2a10c147] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:35:00,018-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2a10c147] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f1efb8b
>2018-05-29 11:35:00,020-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2a10c147] FINISH, SetVdsStatusVDSCommand, log id: 7f1efb8b
>2018-05-29 11:35:00,022-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2a10c147] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:35:00,023-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2a10c147] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:35:00,023-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [2a10c147] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:35:00,028-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [44885b88] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:35:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [44885b88] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:35:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [44885b88] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:35:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [44885b88] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:35:00,036-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [44885b88] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 65d98975
>2018-05-29 11:35:00,038-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [44885b88] FINISH, SetVdsStatusVDSCommand, log id: 65d98975
>2018-05-29 11:35:00,040-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [44885b88] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 74e96910
>2018-05-29 11:35:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [44885b88] FINISH, SetHaMaintenanceModeVDSCommand, log id: 74e96910
>2018-05-29 11:35:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [44885b88] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:35:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [44885b88] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:35:02,069-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6f09751b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 698f0649
>2018-05-29 11:35:02,071-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6f09751b] FINISH, GetHardwareInfoVDSCommand, log id: 698f0649
>2018-05-29 11:35:02,083-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2aa13116] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:35:02,088-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [6e719975] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:35:02,093-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6632127f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:35:02,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6632127f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5550525c
>2018-05-29 11:35:02,474-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6632127f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:35:02,474-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6632127f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5550525c
>2018-05-29 11:35:02,478-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6632127f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:35:02,479-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6632127f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:35:02,483-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6632127f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:35:02,483-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6632127f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:35:02,488-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6632127f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:35:02,488-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6632127f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:35:02,492-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6632127f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:35:02,495-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6632127f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6632127f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:35:02,501-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 63fe5e34
>2018-05-29 11:35:02,501-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:35:02,600-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:35:02,600-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:35:02,600-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:35:02,600-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:35:02,600-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 63fe5e34
>2018-05-29 11:35:02,600-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:35:02,613-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [17e9528f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:35:02,615-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [17e9528f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54f4f4f0
>2018-05-29 11:35:02,617-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [17e9528f] FINISH, SetVdsStatusVDSCommand, log id: 54f4f4f0
>2018-05-29 11:35:02,648-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [17e9528f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 17e9528f, Job ID: 910292cf-aa77-43cd-a1c0-96144d35a942, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:35:02,653-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [17e9528f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6e719975, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:35:02,655-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [17e9528f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6e719975, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:35:02,665-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [25ee9ddf] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:35:02,666-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [25ee9ddf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:35:02,991-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [46f5c1bb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 21eec82e
>2018-05-29 11:35:02,992-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [46f5c1bb] FINISH, GetHardwareInfoVDSCommand, log id: 21eec82e
>2018-05-29 11:35:03,005-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [337f2d8b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:35:03,010-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [aac82c6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:35:03,015-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [66978c31] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:35:03,022-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [66978c31] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 11906d51
>2018-05-29 11:35:03,402-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [66978c31] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:35:03,402-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [66978c31] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 11906d51
>2018-05-29 11:35:03,406-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [66978c31] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:35:03,407-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [66978c31] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:35:03,411-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [66978c31] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:35:03,411-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [66978c31] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:35:03,416-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [66978c31] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:35:03,416-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [66978c31] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:35:03,419-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [66978c31] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:35:03,422-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [66978c31] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 66978c31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:35:03,425-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 46758fce
>2018-05-29 11:35:03,425-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:35:03,522-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:35:03,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:35:03,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:35:03,522-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:35:03,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 46758fce
>2018-05-29 11:35:03,522-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:35:03,536-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [f7453fd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:35:03,537-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [f7453fd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30cc1e3b
>2018-05-29 11:35:03,539-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [f7453fd] FINISH, SetVdsStatusVDSCommand, log id: 30cc1e3b
>2018-05-29 11:35:03,545-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [f7453fd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: f7453fd, Job ID: 3b3fec15-7aa2-4467-b981-d09ed8b5ea07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:35:03,550-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [f7453fd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: aac82c6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:35:03,552-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [f7453fd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: aac82c6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:35:03,563-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [2c75b37f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:35:03,564-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [2c75b37f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [75ea257d] Autorecovering 2 hosts
>2018-05-29 11:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [75ea257d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:40:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1263edf6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1263edf6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1263edf6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1263edf6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:40:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1263edf6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1646240b
>2018-05-29 11:40:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1263edf6] FINISH, SetVdsStatusVDSCommand, log id: 1646240b
>2018-05-29 11:40:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1263edf6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:40:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1263edf6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:40:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [1263edf6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:40:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [25e3e1f4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [25e3e1f4] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [25e3e1f4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [25e3e1f4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:40:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [25e3e1f4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 634cf1b4
>2018-05-29 11:40:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [25e3e1f4] FINISH, SetVdsStatusVDSCommand, log id: 634cf1b4
>2018-05-29 11:40:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [25e3e1f4] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 59b3da2b
>2018-05-29 11:40:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [25e3e1f4] FINISH, SetHaMaintenanceModeVDSCommand, log id: 59b3da2b
>2018-05-29 11:40:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [25e3e1f4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:40:00,102-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [25e3e1f4] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 25e3e1f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-29 11:40:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [25e3e1f4] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:40:00,412-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6385a0ec] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 473f0bd8
>2018-05-29 11:40:00,414-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6385a0ec] FINISH, GetHardwareInfoVDSCommand, log id: 473f0bd8
>2018-05-29 11:40:00,428-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [5e209bc5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:40:00,436-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [304d9383] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:40:00,441-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [49769d36] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:40:00,452-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [49769d36] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3709e760
>2018-05-29 11:40:00,833-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [49769d36] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:40:00,833-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [49769d36] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3709e760
>2018-05-29 11:40:00,836-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [49769d36] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:40:00,837-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [49769d36] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:40:00,841-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [49769d36] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:40:00,842-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [49769d36] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:40:00,846-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [49769d36] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:40:00,846-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [49769d36] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:40:00,849-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [49769d36] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:40:00,851-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [49769d36] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 49769d36, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:40:00,858-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 22253af2
>2018-05-29 11:40:00,858-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:40:00,956-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:40:00,956-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:40:00,956-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:40:00,956-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:40:00,956-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 22253af2
>2018-05-29 11:40:00,956-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:40:00,969-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [6b1efa68] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:40:00,971-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6b1efa68] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 14629bdb
>2018-05-29 11:40:00,973-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6b1efa68] FINISH, SetVdsStatusVDSCommand, log id: 14629bdb
>2018-05-29 11:40:01,004-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6b1efa68] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6b1efa68, Job ID: 7fe9d545-645c-49a0-a27d-6b8e3e14068f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:40:01,008-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6b1efa68] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 304d9383, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:40:01,010-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6b1efa68] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 304d9383, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:40:01,021-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [1a60b205] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:40:01,022-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [1a60b205] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:40:01,191-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5fc850d7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 571a9e37
>2018-05-29 11:40:01,193-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5fc850d7] FINISH, GetHardwareInfoVDSCommand, log id: 571a9e37
>2018-05-29 11:40:01,205-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [633f8ad9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:40:01,209-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [2aeb27ab] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:40:01,214-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [16e73925] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:40:01,221-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [16e73925] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3d496fae
>2018-05-29 11:40:01,602-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16e73925] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:40:01,602-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [16e73925] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3d496fae
>2018-05-29 11:40:01,606-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16e73925] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:40:01,607-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [16e73925] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:40:01,611-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16e73925] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:40:01,612-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [16e73925] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:40:01,616-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16e73925] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:40:01,616-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [16e73925] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:40:01,620-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [16e73925] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:40:01,622-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16e73925] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 16e73925, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:40:01,625-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 160e2825
>2018-05-29 11:40:01,625-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:40:01,720-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:40:01,720-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:40:01,720-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:40:01,720-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:40:01,720-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 160e2825
>2018-05-29 11:40:01,720-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:40:01,733-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [5e53f38c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:40:01,734-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5e53f38c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5b39ae52
>2018-05-29 11:40:01,736-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5e53f38c] FINISH, SetVdsStatusVDSCommand, log id: 5b39ae52
>2018-05-29 11:40:01,742-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e53f38c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5e53f38c, Job ID: 4d67eabb-a1d2-475a-a1ef-c23dd79940ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:40:01,747-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e53f38c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2aeb27ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:40:01,749-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e53f38c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2aeb27ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:40:01,759-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [6b0a708f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:40:01,760-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [6b0a708f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1f2a9a43] Autorecovering 2 hosts
>2018-05-29 11:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1f2a9a43] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:45:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [c86477d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [c86477d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [c86477d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [c86477d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:45:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [c86477d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3601803b
>2018-05-29 11:45:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [c86477d] FINISH, SetVdsStatusVDSCommand, log id: 3601803b
>2018-05-29 11:45:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [c86477d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:45:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [c86477d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:45:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [c86477d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:45:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4dde0f6a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:45:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4dde0f6a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:45:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4dde0f6a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:45:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4dde0f6a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:45:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4dde0f6a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d03eccc
>2018-05-29 11:45:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4dde0f6a] FINISH, SetVdsStatusVDSCommand, log id: 2d03eccc
>2018-05-29 11:45:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [4dde0f6a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 42870835
>2018-05-29 11:45:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [4dde0f6a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 42870835
>2018-05-29 11:45:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4dde0f6a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:45:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4dde0f6a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:45:01,672-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [47e5bbbc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1e718fab
>2018-05-29 11:45:01,673-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [47e5bbbc] FINISH, GetHardwareInfoVDSCommand, log id: 1e718fab
>2018-05-29 11:45:01,686-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [2ad096a9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:45:01,694-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [167e6e43] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:45:01,699-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7060438d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:45:01,711-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7060438d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1ca59fb
>2018-05-29 11:45:02,084-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7060438d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:45:02,084-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7060438d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1ca59fb
>2018-05-29 11:45:02,088-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7060438d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:45:02,088-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7060438d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:45:02,092-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7060438d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:45:02,093-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7060438d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:45:02,096-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7060438d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:45:02,097-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7060438d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:45:02,100-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7060438d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:45:02,103-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7060438d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7060438d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:45:02,109-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2c5e7c9
>2018-05-29 11:45:02,109-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:45:02,210-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:45:02,210-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:45:02,210-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:45:02,210-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:45:02,210-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2c5e7c9
>2018-05-29 11:45:02,210-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:45:02,224-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [79cdd8fd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:45:02,225-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [79cdd8fd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 14975fb2
>2018-05-29 11:45:02,227-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [79cdd8fd] FINISH, SetVdsStatusVDSCommand, log id: 14975fb2
>2018-05-29 11:45:02,258-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [79cdd8fd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 79cdd8fd, Job ID: 0734a900-59f0-4c4c-8197-75d468a06275, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:45:02,263-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [79cdd8fd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 167e6e43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:45:02,265-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [79cdd8fd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 167e6e43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:45:02,276-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [709469ba] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:45:02,277-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [709469ba] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:45:02,377-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5e49ddd5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3dec8dae
>2018-05-29 11:45:02,378-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5e49ddd5] FINISH, GetHardwareInfoVDSCommand, log id: 3dec8dae
>2018-05-29 11:45:02,390-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [5587734d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:45:02,394-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [5e1282b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:45:02,399-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [3bfa770f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:45:02,406-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [3bfa770f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 191870b4
>2018-05-29 11:45:02,784-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3bfa770f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:45:02,785-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [3bfa770f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 191870b4
>2018-05-29 11:45:02,788-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3bfa770f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:45:02,789-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [3bfa770f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:45:02,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3bfa770f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:45:02,793-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [3bfa770f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:45:02,797-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3bfa770f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:45:02,798-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [3bfa770f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:45:02,800-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [3bfa770f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:45:02,803-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3bfa770f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3bfa770f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:45:02,806-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7f87541e
>2018-05-29 11:45:02,806-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:45:02,905-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:45:02,905-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:45:02,905-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:45:02,905-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:45:02,905-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7f87541e
>2018-05-29 11:45:02,905-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:45:02,917-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [398d0fe] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:45:02,918-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [398d0fe] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7fc92018
>2018-05-29 11:45:02,920-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [398d0fe] FINISH, SetVdsStatusVDSCommand, log id: 7fc92018
>2018-05-29 11:45:02,926-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [398d0fe] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 398d0fe, Job ID: be29038a-7fda-4443-b4fe-09e53df8dc32, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:45:02,930-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [398d0fe] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5e1282b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:45:02,932-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [398d0fe] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5e1282b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:45:02,942-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [dc7bb76] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:45:02,944-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [dc7bb76] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [fbffba2] Autorecovering 2 hosts
>2018-05-29 11:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [fbffba2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:50:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d921e8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d921e8] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d921e8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d921e8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:50:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3d921e8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3e579e2d
>2018-05-29 11:50:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3d921e8] FINISH, SetVdsStatusVDSCommand, log id: 3e579e2d
>2018-05-29 11:50:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d921e8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:50:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d921e8] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:50:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3d921e8] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:50:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1925461e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:50:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1925461e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:50:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1925461e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:50:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1925461e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:50:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1925461e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3aeddc4d
>2018-05-29 11:50:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1925461e] FINISH, SetVdsStatusVDSCommand, log id: 3aeddc4d
>2018-05-29 11:50:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [1925461e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 45ae9698
>2018-05-29 11:50:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [1925461e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 45ae9698
>2018-05-29 11:50:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1925461e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:50:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1925461e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:50:00,569-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [13983791] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7f225951
>2018-05-29 11:50:00,570-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [13983791] FINISH, GetHardwareInfoVDSCommand, log id: 7f225951
>2018-05-29 11:50:00,584-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [29324ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:50:00,592-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [9b574cb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:50:00,598-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [62e72956] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:50:00,662-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [62e72956] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7e9b271
>2018-05-29 11:50:01,050-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [62e72956] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:50:01,050-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [62e72956] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7e9b271
>2018-05-29 11:50:01,054-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [62e72956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:50:01,054-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [62e72956] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:50:01,058-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [62e72956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:50:01,059-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [62e72956] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:50:01,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [62e72956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:50:01,064-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [62e72956] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:50:01,066-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [62e72956] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:50:01,069-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [62e72956] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 62e72956, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:50:01,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 45b41aa8
>2018-05-29 11:50:01,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:50:01,176-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:50:01,176-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:50:01,176-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:50:01,176-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:50:01,176-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 45b41aa8
>2018-05-29 11:50:01,176-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:50:01,191-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [3b01ada9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:50:01,193-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3b01ada9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 230caca2
>2018-05-29 11:50:01,195-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3b01ada9] FINISH, SetVdsStatusVDSCommand, log id: 230caca2
>2018-05-29 11:50:01,227-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3b01ada9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3b01ada9, Job ID: facfbfd8-e355-4ae3-9694-7513e110bb03, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:50:01,232-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3b01ada9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 9b574cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:50:01,234-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3b01ada9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 9b574cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:50:01,245-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [5086b37] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:50:01,246-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [5086b37] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:50:02,928-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [496347f6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4f272bf3
>2018-05-29 11:50:02,930-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [496347f6] FINISH, GetHardwareInfoVDSCommand, log id: 4f272bf3
>2018-05-29 11:50:02,944-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [55ebb8af] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:50:02,948-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6558f0d6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:50:02,953-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [28daa5a6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:50:02,961-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [28daa5a6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 709a28de
>2018-05-29 11:50:05,272-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [28daa5a6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:50:05,272-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [28daa5a6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 709a28de
>2018-05-29 11:50:05,276-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [28daa5a6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:50:05,277-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [28daa5a6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:50:05,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [28daa5a6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:50:05,281-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [28daa5a6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:50:05,285-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [28daa5a6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:50:05,286-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [28daa5a6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:50:05,290-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [28daa5a6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:50:05,293-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [28daa5a6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 28daa5a6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:50:05,296-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2162651e
>2018-05-29 11:50:05,296-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:50:05,395-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:50:05,395-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:50:05,395-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:50:05,395-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:50:05,395-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2162651e
>2018-05-29 11:50:05,395-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:50:05,409-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [6327a986] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:50:05,410-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6327a986] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 325e303d
>2018-05-29 11:50:05,412-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6327a986] FINISH, SetVdsStatusVDSCommand, log id: 325e303d
>2018-05-29 11:50:05,418-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6327a986] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6327a986, Job ID: 98ae14ab-1bae-45ad-9566-d8618b0bc2b5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:50:05,422-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6327a986] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6558f0d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:50:05,424-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6327a986] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6558f0d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:50:05,435-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [51fc95da] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:50:05,436-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [51fc95da] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:54:43,003-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler6) [785a95ba] Attempting to update VMs/Templates Ovf.
>2018-05-29 11:54:43,006-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 11:54:43,006-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 11:54:43,006-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:54:43,010-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 11:54:43,014-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 11:54:43,014-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 11:54:43,014-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 11:54:43,014-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 11:54:43,015-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 11:54:43,015-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 11:55:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [758d8d59] Autorecovering 2 hosts
>2018-05-29 11:55:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [758d8d59] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:55:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2af7b478] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2af7b478] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2af7b478] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2af7b478] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:55:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2af7b478] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 764e28be
>2018-05-29 11:55:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2af7b478] FINISH, SetVdsStatusVDSCommand, log id: 764e28be
>2018-05-29 11:55:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2af7b478] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:55:00,079-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2af7b478] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 2af7b478, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-29 11:55:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2af7b478] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 11:55:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2af7b478] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:55:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13e3aaa8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:55:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13e3aaa8] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 11:55:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13e3aaa8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:55:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13e3aaa8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:55:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [13e3aaa8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5541058
>2018-05-29 11:55:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [13e3aaa8] FINISH, SetVdsStatusVDSCommand, log id: 5541058
>2018-05-29 11:55:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [13e3aaa8] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2550d3d
>2018-05-29 11:55:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [13e3aaa8] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2550d3d
>2018-05-29 11:55:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13e3aaa8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 11:55:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13e3aaa8] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 11:55:01,897-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6ecb2358] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 168a0263
>2018-05-29 11:55:01,898-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6ecb2358] FINISH, GetHardwareInfoVDSCommand, log id: 168a0263
>2018-05-29 11:55:01,912-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [458ab0fa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:55:01,921-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [545a6061] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:55:01,926-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [42bba5da] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:55:01,939-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [42bba5da] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 25e0ecf9
>2018-05-29 11:55:02,330-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42bba5da] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:55:02,330-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [42bba5da] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 25e0ecf9
>2018-05-29 11:55:02,334-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42bba5da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:55:02,335-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [42bba5da] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:55:02,338-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42bba5da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:55:02,339-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [42bba5da] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:55:02,343-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42bba5da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:55:02,344-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [42bba5da] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:55:02,346-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [42bba5da] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:55:02,349-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42bba5da] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 42bba5da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:55:02,355-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 47da0b41
>2018-05-29 11:55:02,355-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:55:02,454-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:55:02,454-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:55:02,454-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:55:02,454-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:55:02,454-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 47da0b41
>2018-05-29 11:55:02,454-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:55:02,467-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [458d5b57] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:55:02,468-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [458d5b57] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f7d18ec
>2018-05-29 11:55:02,471-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [458d5b57] FINISH, SetVdsStatusVDSCommand, log id: 7f7d18ec
>2018-05-29 11:55:02,503-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [458d5b57] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 458d5b57, Job ID: 029a4d24-59f1-4b30-b58b-43b6eb1e01f1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:55:02,507-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [458d5b57] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 545a6061, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:55:02,509-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [458d5b57] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 545a6061, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:55:02,520-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [2fc0a9ec] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 11:55:02,521-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [2fc0a9ec] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 11:55:03,192-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [586ab90a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 50b45453
>2018-05-29 11:55:03,194-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [586ab90a] FINISH, GetHardwareInfoVDSCommand, log id: 50b45453
>2018-05-29 11:55:03,206-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [2064105f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:55:03,210-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [2b2f1691] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:55:03,215-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [280b2ffc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 11:55:03,222-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [280b2ffc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 41cf4358
>2018-05-29 11:55:03,591-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [280b2ffc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 11:55:03,591-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [280b2ffc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 41cf4358
>2018-05-29 11:55:03,595-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [280b2ffc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:55:03,596-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [280b2ffc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:55:03,599-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [280b2ffc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:55:03,600-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [280b2ffc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:55:03,604-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [280b2ffc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 11:55:03,605-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [280b2ffc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 11:55:03,607-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [280b2ffc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 11:55:03,610-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [280b2ffc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 280b2ffc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 11:55:03,613-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5561ba49
>2018-05-29 11:55:03,613-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 11:55:03,710-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:55:03,710-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 11:55:03,710-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 11:55:03,710-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 11:55:03,710-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5561ba49
>2018-05-29 11:55:03,710-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 11:55:03,723-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [623acf29] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:55:03,724-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [623acf29] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f6a470d
>2018-05-29 11:55:03,726-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [623acf29] FINISH, SetVdsStatusVDSCommand, log id: 7f6a470d
>2018-05-29 11:55:03,731-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [623acf29] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 623acf29, Job ID: c93a1298-4ac9-4b42-8d7b-84788763c6bd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 11:55:03,736-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [623acf29] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2b2f1691, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 11:55:03,738-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [623acf29] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2b2f1691, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 11:55:03,748-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7f2acba6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 11:55:03,749-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7f2acba6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1ab77b05] Autorecovering 2 hosts
>2018-05-29 12:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1ab77b05] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:00:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2f0a8315] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2f0a8315] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2f0a8315] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2f0a8315] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:00:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2f0a8315] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39e93de5
>2018-05-29 12:00:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2f0a8315] FINISH, SetVdsStatusVDSCommand, log id: 39e93de5
>2018-05-29 12:00:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2f0a8315] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:00:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2f0a8315] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:00:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [2f0a8315] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:00:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3e670de7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3e670de7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3e670de7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3e670de7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:00:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3e670de7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 109b49ed
>2018-05-29 12:00:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3e670de7] FINISH, SetVdsStatusVDSCommand, log id: 109b49ed
>2018-05-29 12:00:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [3e670de7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 65076c55
>2018-05-29 12:00:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [3e670de7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 65076c55
>2018-05-29 12:00:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3e670de7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:00:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3e670de7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:00:01,427-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [683e8200] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 60a53ff9
>2018-05-29 12:00:01,429-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [683e8200] FINISH, GetHardwareInfoVDSCommand, log id: 60a53ff9
>2018-05-29 12:00:01,442-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [2ead6570] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:00:01,447-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [4903338] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:00:01,452-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7a3cb68d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:00:01,459-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7a3cb68d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 33d11a66
>2018-05-29 12:00:01,825-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a3cb68d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:00:01,825-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7a3cb68d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 33d11a66
>2018-05-29 12:00:01,829-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a3cb68d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:00:01,830-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7a3cb68d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:00:01,834-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a3cb68d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:00:01,834-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7a3cb68d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:00:01,838-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a3cb68d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:00:01,839-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7a3cb68d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:00:01,842-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7a3cb68d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:00:01,844-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a3cb68d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7a3cb68d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:00:01,847-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5961861e
>2018-05-29 12:00:01,847-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:00:01,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:00:01,944-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:00:01,944-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:00:01,944-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:00:01,944-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5961861e
>2018-05-29 12:00:01,944-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:00:01,957-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [380fb947] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:00:01,958-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [380fb947] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1fdf6436
>2018-05-29 12:00:01,960-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [380fb947] FINISH, SetVdsStatusVDSCommand, log id: 1fdf6436
>2018-05-29 12:00:01,989-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [380fb947] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 380fb947, Job ID: f3a0434d-6f12-4df2-9742-38c323a003ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:00:01,993-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [380fb947] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4903338, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:00:01,995-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [380fb947] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4903338, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:00:02,006-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [2783d80c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:00:02,007-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2783d80c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:00:03,963-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6f40738a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6738c5d6
>2018-05-29 12:00:03,965-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6f40738a] FINISH, GetHardwareInfoVDSCommand, log id: 6738c5d6
>2018-05-29 12:00:03,978-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [44bcc85a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:00:03,982-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [617bf27c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:00:03,987-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [1f134f35] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:00:03,994-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [1f134f35] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f4a9821
>2018-05-29 12:00:04,365-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1f134f35] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:00:04,365-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [1f134f35] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6f4a9821
>2018-05-29 12:00:04,369-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1f134f35] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:00:04,370-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [1f134f35] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:00:04,374-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1f134f35] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:00:04,374-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [1f134f35] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:00:04,378-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1f134f35] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:00:04,379-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [1f134f35] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:00:04,382-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [1f134f35] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:00:04,384-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1f134f35] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1f134f35, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:00:04,387-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4c22325
>2018-05-29 12:00:04,387-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:00:04,482-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:00:04,483-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:00:04,483-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:00:04,483-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:00:04,483-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4c22325
>2018-05-29 12:00:04,483-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:00:04,496-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [108d94f5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:00:04,497-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [108d94f5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1ed5ebf9
>2018-05-29 12:00:04,499-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [108d94f5] FINISH, SetVdsStatusVDSCommand, log id: 1ed5ebf9
>2018-05-29 12:00:04,504-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [108d94f5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 108d94f5, Job ID: de3dc367-257b-43c1-a9ce-122661981828, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:00:04,509-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [108d94f5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 617bf27c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:00:04,511-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [108d94f5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 617bf27c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:00:04,522-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [112929ee] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:00:04,523-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [112929ee] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:05:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [993dc5e] Autorecovering 2 hosts
>2018-05-29 12:05:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [993dc5e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:05:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [737c6699] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:05:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [737c6699] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:05:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [737c6699] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:05:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [737c6699] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:05:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [737c6699] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26e6b47a
>2018-05-29 12:05:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [737c6699] FINISH, SetVdsStatusVDSCommand, log id: 26e6b47a
>2018-05-29 12:05:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [737c6699] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:05:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [737c6699] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:05:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [737c6699] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:05:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [69ad19df] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:05:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [69ad19df] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:05:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [69ad19df] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:05:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [69ad19df] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:05:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [69ad19df] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 386fd29c
>2018-05-29 12:05:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [69ad19df] FINISH, SetVdsStatusVDSCommand, log id: 386fd29c
>2018-05-29 12:05:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [69ad19df] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: c44a95f
>2018-05-29 12:05:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [69ad19df] FINISH, SetHaMaintenanceModeVDSCommand, log id: c44a95f
>2018-05-29 12:05:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [69ad19df] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:05:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [69ad19df] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:05:02,120-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1bd32ad0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 65f0a6ff
>2018-05-29 12:05:02,121-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1bd32ad0] FINISH, GetHardwareInfoVDSCommand, log id: 65f0a6ff
>2018-05-29 12:05:02,135-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [53a4f93f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:05:02,140-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [102bcfb0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:05:02,145-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [43f48b43] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:05:02,157-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [43f48b43] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: d23c33a
>2018-05-29 12:05:02,542-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [43f48b43] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:05:02,542-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [43f48b43] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: d23c33a
>2018-05-29 12:05:02,546-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [43f48b43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:05:02,547-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [43f48b43] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:05:02,551-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [43f48b43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:05:02,551-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [43f48b43] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:05:02,555-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [43f48b43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:05:02,556-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [43f48b43] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:05:02,559-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [43f48b43] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:05:02,561-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [43f48b43] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 43f48b43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:05:02,568-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7fde3c9
>2018-05-29 12:05:02,568-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:05:02,636-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [10338438] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1cce29ac
>2018-05-29 12:05:02,638-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [10338438] FINISH, GetHardwareInfoVDSCommand, log id: 1cce29ac
>2018-05-29 12:05:02,650-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4e2a1e12] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:05:02,655-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [637c6e0a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:05:02,660-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [71c81af9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:05:02,666-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [71c81af9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f0fb3d
>2018-05-29 12:05:02,670-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:05:02,670-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:05:02,670-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:05:02,670-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:05:02,670-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7fde3c9
>2018-05-29 12:05:02,670-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:05:02,682-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [5acea721] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:05:02,683-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5acea721] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 9354fa0
>2018-05-29 12:05:02,685-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5acea721] FINISH, SetVdsStatusVDSCommand, log id: 9354fa0
>2018-05-29 12:05:02,693-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5acea721] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5acea721, Job ID: d3819187-1d71-4c1f-8340-bb0c40db2e6d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:05:02,698-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5acea721] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 102bcfb0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:05:02,700-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5acea721] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 102bcfb0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:05:02,711-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [783ca7bb] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:05:02,712-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [783ca7bb] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:05:03,033-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c81af9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:05:03,033-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [71c81af9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f0fb3d
>2018-05-29 12:05:03,037-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c81af9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:05:03,038-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [71c81af9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:05:03,041-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c81af9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:05:03,042-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [71c81af9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:05:03,046-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c81af9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:05:03,047-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [71c81af9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:05:03,050-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [71c81af9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:05:03,053-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c81af9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 71c81af9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:05:03,055-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 74f2ddd4
>2018-05-29 12:05:03,055-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:05:03,153-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:05:03,153-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:05:03,153-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:05:03,153-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:05:03,153-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 74f2ddd4
>2018-05-29 12:05:03,153-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:05:03,165-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [229aa01b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:05:03,166-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [229aa01b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1d8be7a2
>2018-05-29 12:05:03,168-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [229aa01b] FINISH, SetVdsStatusVDSCommand, log id: 1d8be7a2
>2018-05-29 12:05:03,173-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [229aa01b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 229aa01b, Job ID: bef77851-5681-4ef0-a641-6ce9e8f87d2c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:05:03,178-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [229aa01b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 637c6e0a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:05:03,180-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [229aa01b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 637c6e0a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:05:03,190-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [1260fde0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:05:03,191-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [1260fde0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5ae296d2] Autorecovering 2 hosts
>2018-05-29 12:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5ae296d2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:10:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [413e43a0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [413e43a0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [413e43a0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [413e43a0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:10:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [413e43a0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: c11b0d0
>2018-05-29 12:10:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [413e43a0] FINISH, SetVdsStatusVDSCommand, log id: c11b0d0
>2018-05-29 12:10:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [413e43a0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:10:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [413e43a0] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:10:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [413e43a0] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:10:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2de024c1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:10:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2de024c1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:10:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2de024c1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:10:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2de024c1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:10:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2de024c1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2dd1807e
>2018-05-29 12:10:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2de024c1] FINISH, SetVdsStatusVDSCommand, log id: 2dd1807e
>2018-05-29 12:10:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [2de024c1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 732ee0c5
>2018-05-29 12:10:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [2de024c1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 732ee0c5
>2018-05-29 12:10:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2de024c1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:10:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2de024c1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:10:00,485-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2fd5aff] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2013359c
>2018-05-29 12:10:00,487-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2fd5aff] FINISH, GetHardwareInfoVDSCommand, log id: 2013359c
>2018-05-29 12:10:00,500-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [c7d7abc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:10:00,508-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [30172e38] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:10:00,513-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [53eb145b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:10:00,520-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [53eb145b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 352cfd2e
>2018-05-29 12:10:00,812-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [734d7d01] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5ed92562
>2018-05-29 12:10:00,814-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [734d7d01] FINISH, GetHardwareInfoVDSCommand, log id: 5ed92562
>2018-05-29 12:10:00,827-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [10e6300c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:10:00,832-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [7d0ea3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:10:00,837-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [14225c0f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:10:00,844-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [14225c0f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 47784b5e
>2018-05-29 12:10:00,895-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [53eb145b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:10:00,895-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [53eb145b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 352cfd2e
>2018-05-29 12:10:00,899-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [53eb145b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:10:00,900-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [53eb145b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:10:00,903-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [53eb145b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:10:00,904-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [53eb145b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:10:00,908-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [53eb145b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:10:00,909-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [53eb145b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:10:00,911-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [53eb145b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:10:00,914-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [53eb145b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 53eb145b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:10:00,917-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7732b4c3
>2018-05-29 12:10:00,917-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:10:01,013-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-10) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:10:01,014-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:10:01,014-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:10:01,014-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:10:01,014-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7732b4c3
>2018-05-29 12:10:01,014-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-10) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:10:01,027-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [5d6b417c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:10:01,028-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5d6b417c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b6c80f1
>2018-05-29 12:10:01,030-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5d6b417c] FINISH, SetVdsStatusVDSCommand, log id: 1b6c80f1
>2018-05-29 12:10:01,062-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5d6b417c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5d6b417c, Job ID: 28136b99-0c76-485d-8d8d-d204bbb3cc4b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:10:01,067-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5d6b417c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 30172e38, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:10:01,069-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5d6b417c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 30172e38, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:10:01,080-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [488cdfef] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:10:01,081-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [488cdfef] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:10:01,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [14225c0f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:10:01,218-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [14225c0f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 47784b5e
>2018-05-29 12:10:01,222-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [14225c0f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:10:01,223-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [14225c0f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:10:01,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [14225c0f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:10:01,227-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [14225c0f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:10:01,231-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [14225c0f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:10:01,232-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [14225c0f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:10:01,235-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [14225c0f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:10:01,237-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [14225c0f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 14225c0f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:10:01,240-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3aa4f029
>2018-05-29 12:10:01,240-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:10:01,339-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:10:01,339-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:10:01,339-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:10:01,339-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:10:01,339-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3aa4f029
>2018-05-29 12:10:01,339-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:10:01,352-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [221d2b70] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:10:01,353-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [221d2b70] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 56866361
>2018-05-29 12:10:01,355-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [221d2b70] FINISH, SetVdsStatusVDSCommand, log id: 56866361
>2018-05-29 12:10:01,361-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [221d2b70] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 221d2b70, Job ID: aba05037-398d-458d-8c36-65f9edf42f3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:10:01,365-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [221d2b70] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7d0ea3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:10:01,367-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [221d2b70] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7d0ea3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:10:01,378-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [1735156c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:10:01,379-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [1735156c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [c889a30] Autorecovering 2 hosts
>2018-05-29 12:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [c889a30] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:15:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [416eb396] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [416eb396] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [416eb396] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [416eb396] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:15:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [416eb396] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3237f29a
>2018-05-29 12:15:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [416eb396] FINISH, SetVdsStatusVDSCommand, log id: 3237f29a
>2018-05-29 12:15:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [416eb396] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:15:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [416eb396] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:15:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [416eb396] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:15:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [618689fc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:15:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [618689fc] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:15:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [618689fc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:15:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [618689fc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:15:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [618689fc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 508b4501
>2018-05-29 12:15:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [618689fc] FINISH, SetVdsStatusVDSCommand, log id: 508b4501
>2018-05-29 12:15:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [618689fc] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 71405537
>2018-05-29 12:15:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [618689fc] FINISH, SetHaMaintenanceModeVDSCommand, log id: 71405537
>2018-05-29 12:15:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [618689fc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:15:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [618689fc] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:15:01,748-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [50f8348c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 78a854a7
>2018-05-29 12:15:01,750-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [50f8348c] FINISH, GetHardwareInfoVDSCommand, log id: 78a854a7
>2018-05-29 12:15:01,763-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [7a2360e5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:15:01,768-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [3b92d40e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:15:01,773-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [201a4ac9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:15:01,830-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [201a4ac9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 24aad26
>2018-05-29 12:15:02,014-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1b48ab50] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 76c791b5
>2018-05-29 12:15:02,016-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1b48ab50] FINISH, GetHardwareInfoVDSCommand, log id: 76c791b5
>2018-05-29 12:15:02,028-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2b14a755] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:15:02,033-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4de88ef5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:15:02,038-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6b03b8b0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:15:02,044-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6b03b8b0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4a4aaa8c
>2018-05-29 12:15:02,214-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [201a4ac9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:15:02,214-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [201a4ac9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 24aad26
>2018-05-29 12:15:02,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [201a4ac9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:15:02,219-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [201a4ac9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:15:02,222-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [201a4ac9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:15:02,223-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [201a4ac9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:15:02,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [201a4ac9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:15:02,227-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [201a4ac9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:15:02,230-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [201a4ac9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:15:02,232-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [201a4ac9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 201a4ac9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:15:02,239-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1145060
>2018-05-29 12:15:02,239-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:15:02,338-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:15:02,338-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:15:02,338-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:15:02,338-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:15:02,339-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1145060
>2018-05-29 12:15:02,339-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:15:02,351-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [6d5ed5c7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:15:02,353-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6d5ed5c7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 137b1da
>2018-05-29 12:15:02,355-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6d5ed5c7] FINISH, SetVdsStatusVDSCommand, log id: 137b1da
>2018-05-29 12:15:02,363-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6d5ed5c7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6d5ed5c7, Job ID: bf7a8e96-1e4c-4c2f-bc46-d6f2e66a8d97, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:15:02,367-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6d5ed5c7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b92d40e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:15:02,369-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6d5ed5c7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b92d40e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:15:02,380-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [53ec8626] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:15:02,381-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [53ec8626] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:15:02,404-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b03b8b0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:15:02,404-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6b03b8b0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4a4aaa8c
>2018-05-29 12:15:02,408-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b03b8b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:15:02,409-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6b03b8b0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:15:02,413-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b03b8b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:15:02,414-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6b03b8b0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:15:02,418-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b03b8b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:15:02,418-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6b03b8b0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:15:02,421-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6b03b8b0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:15:02,424-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b03b8b0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6b03b8b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:15:02,427-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4bae770
>2018-05-29 12:15:02,427-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:15:02,522-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:15:02,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:15:02,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:15:02,522-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:15:02,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4bae770
>2018-05-29 12:15:02,522-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:15:02,535-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [16f855e6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:15:02,536-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [16f855e6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 698f942b
>2018-05-29 12:15:02,538-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [16f855e6] FINISH, SetVdsStatusVDSCommand, log id: 698f942b
>2018-05-29 12:15:02,544-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [16f855e6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 16f855e6, Job ID: 455f60e1-8c4d-4bc3-8296-24baa60202fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:15:02,548-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [16f855e6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4de88ef5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:15:02,550-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [16f855e6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4de88ef5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:15:02,561-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [37b67e16] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:15:02,562-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [37b67e16] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:20:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [18d97500] Autorecovering 2 hosts
>2018-05-29 12:20:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [18d97500] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:20:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1a3f5dda] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:20:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1a3f5dda] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:20:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1a3f5dda] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:20:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1a3f5dda] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:20:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1a3f5dda] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61b52d74
>2018-05-29 12:20:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1a3f5dda] FINISH, SetVdsStatusVDSCommand, log id: 61b52d74
>2018-05-29 12:20:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1a3f5dda] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:20:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1a3f5dda] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:20:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1a3f5dda] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:20:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62b9b9d9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:20:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62b9b9d9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:20:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62b9b9d9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:20:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62b9b9d9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:20:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [62b9b9d9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7534137c
>2018-05-29 12:20:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [62b9b9d9] FINISH, SetVdsStatusVDSCommand, log id: 7534137c
>2018-05-29 12:20:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [62b9b9d9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6acb7327
>2018-05-29 12:20:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [62b9b9d9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6acb7327
>2018-05-29 12:20:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62b9b9d9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:20:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62b9b9d9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:20:02,963-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [6b88039] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 62b2dd5
>2018-05-29 12:20:02,964-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [6b88039] FINISH, GetHardwareInfoVDSCommand, log id: 62b2dd5
>2018-05-29 12:20:02,978-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [116f3666] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:20:02,987-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [9fa88bd] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:20:02,992-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4419d877] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:20:03,004-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4419d877] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 12e675fa
>2018-05-29 12:20:03,221-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [19106d56] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 46368f1d
>2018-05-29 12:20:03,223-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [19106d56] FINISH, GetHardwareInfoVDSCommand, log id: 46368f1d
>2018-05-29 12:20:03,238-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [47bc2f6e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:20:03,243-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [2c828e9e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:20:03,247-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [78e6caed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:20:03,254-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [78e6caed] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: cafb29
>2018-05-29 12:20:03,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4419d877] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:20:03,388-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4419d877] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 12e675fa
>2018-05-29 12:20:03,392-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4419d877] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:20:03,393-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4419d877] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:20:03,397-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4419d877] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:20:03,397-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4419d877] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:20:03,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4419d877] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:20:03,402-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4419d877] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:20:03,404-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4419d877] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:20:03,407-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4419d877] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4419d877, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:20:03,414-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6369e426
>2018-05-29 12:20:03,414-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:20:03,512-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:20:03,512-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:20:03,512-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:20:03,512-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:20:03,512-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6369e426
>2018-05-29 12:20:03,512-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:20:03,525-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [20a47d02] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:20:03,526-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [20a47d02] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 586e1679
>2018-05-29 12:20:03,528-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [20a47d02] FINISH, SetVdsStatusVDSCommand, log id: 586e1679
>2018-05-29 12:20:03,559-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [20a47d02] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 20a47d02, Job ID: 0f532906-5987-43a2-8f40-bc6f0aefdb6f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:20:03,564-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [20a47d02] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 9fa88bd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:20:03,566-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [20a47d02] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 9fa88bd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:20:03,577-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [f38e08c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:20:03,578-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [f38e08c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:20:03,625-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [78e6caed] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:20:03,625-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [78e6caed] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: cafb29
>2018-05-29 12:20:03,629-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [78e6caed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:20:03,629-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [78e6caed] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:20:03,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [78e6caed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:20:03,634-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [78e6caed] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:20:03,638-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [78e6caed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:20:03,638-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [78e6caed] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:20:03,641-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [78e6caed] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:20:03,644-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [78e6caed] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 78e6caed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:20:03,647-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 377f5cfa
>2018-05-29 12:20:03,647-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:20:03,745-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:20:03,745-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:20:03,745-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:20:03,745-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:20:03,745-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 377f5cfa
>2018-05-29 12:20:03,745-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:20:03,758-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [5d8a2e90] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:20:03,759-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5d8a2e90] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 36c91031
>2018-05-29 12:20:03,761-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5d8a2e90] FINISH, SetVdsStatusVDSCommand, log id: 36c91031
>2018-05-29 12:20:03,767-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8a2e90] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5d8a2e90, Job ID: a16fb409-9cbe-4c0e-be56-5ffbc102fe61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:20:03,771-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8a2e90] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2c828e9e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:20:03,773-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8a2e90] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2c828e9e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:20:03,783-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [7710e8f5] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:20:03,784-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [7710e8f5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [24482dcb] Autorecovering 2 hosts
>2018-05-29 12:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [24482dcb] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:25:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [22b9c274] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:25:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [22b9c274] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:25:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [22b9c274] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:25:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [22b9c274] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:25:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [22b9c274] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 55ebc8d6
>2018-05-29 12:25:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [22b9c274] FINISH, SetVdsStatusVDSCommand, log id: 55ebc8d6
>2018-05-29 12:25:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [22b9c274] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:25:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [22b9c274] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:25:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [22b9c274] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:25:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4d552b80] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:25:00,137-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4d552b80] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:25:00,137-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4d552b80] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:25:00,137-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4d552b80] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:25:00,139-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4d552b80] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6211db5e
>2018-05-29 12:25:00,141-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4d552b80] FINISH, SetVdsStatusVDSCommand, log id: 6211db5e
>2018-05-29 12:25:00,144-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4d552b80] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 54077e08
>2018-05-29 12:25:00,146-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4d552b80] FINISH, SetHaMaintenanceModeVDSCommand, log id: 54077e08
>2018-05-29 12:25:00,146-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4d552b80] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:25:00,148-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4d552b80] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:25:01,177-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [498430c7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3688719e
>2018-05-29 12:25:01,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [498430c7] FINISH, GetHardwareInfoVDSCommand, log id: 3688719e
>2018-05-29 12:25:01,192-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7e0237fd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:25:01,200-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [2702321f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:25:01,206-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [a7d2544] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:25:01,217-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [a7d2544] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3ea2bd36
>2018-05-29 12:25:01,421-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7ebacfc6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1905f9d0
>2018-05-29 12:25:01,423-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7ebacfc6] FINISH, GetHardwareInfoVDSCommand, log id: 1905f9d0
>2018-05-29 12:25:01,436-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [2554633b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:25:01,440-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [eb34907] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:25:01,445-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [af1b419] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:25:01,452-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [af1b419] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 20ef7937
>2018-05-29 12:25:01,594-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a7d2544] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:25:01,594-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [a7d2544] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3ea2bd36
>2018-05-29 12:25:01,598-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a7d2544] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:25:01,599-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [a7d2544] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:25:01,602-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a7d2544] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:25:01,603-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [a7d2544] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:25:01,607-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a7d2544] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:25:01,608-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [a7d2544] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:25:01,610-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [a7d2544] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:25:01,613-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a7d2544] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: a7d2544, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:25:01,620-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 49084988
>2018-05-29 12:25:01,620-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:25:01,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:25:01,716-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:25:01,716-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:25:01,716-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:25:01,716-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 49084988
>2018-05-29 12:25:01,716-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:25:01,730-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [33caa6ec] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:25:01,731-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [33caa6ec] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 77830bae
>2018-05-29 12:25:01,733-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [33caa6ec] FINISH, SetVdsStatusVDSCommand, log id: 77830bae
>2018-05-29 12:25:01,765-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [33caa6ec] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 33caa6ec, Job ID: b903b8ed-bbf6-4dbd-9aea-ae07163e44d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:25:01,770-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [33caa6ec] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2702321f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:25:01,772-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [33caa6ec] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2702321f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:25:01,783-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [71c12d85] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:25:01,784-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [71c12d85] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:25:02,818-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [af1b419] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:25:02,818-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [af1b419] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 20ef7937
>2018-05-29 12:25:02,822-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [af1b419] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:25:02,822-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [af1b419] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:25:02,826-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [af1b419] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:25:02,827-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [af1b419] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:25:02,831-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [af1b419] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:25:02,832-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [af1b419] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:25:02,835-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [af1b419] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:25:02,837-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [af1b419] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: af1b419, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:25:02,840-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6d74c080
>2018-05-29 12:25:02,840-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:25:02,936-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:25:02,936-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:25:02,936-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:25:02,936-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:25:02,937-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6d74c080
>2018-05-29 12:25:02,937-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:25:02,950-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [5dec1320] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:25:02,951-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5dec1320] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17a89c1a
>2018-05-29 12:25:02,953-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5dec1320] FINISH, SetVdsStatusVDSCommand, log id: 17a89c1a
>2018-05-29 12:25:02,984-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5dec1320] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5dec1320, Job ID: 1e0aa27d-ce96-4652-91ab-16c1c22f2560, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:25:02,988-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5dec1320] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: eb34907, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:25:02,990-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5dec1320] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: eb34907, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:25:03,001-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [b5274c9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:25:03,002-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [b5274c9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [284865f] Autorecovering 2 hosts
>2018-05-29 12:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [284865f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:30:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7f8212e6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7f8212e6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7f8212e6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7f8212e6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:30:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7f8212e6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b7e38e1
>2018-05-29 12:30:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7f8212e6] FINISH, SetVdsStatusVDSCommand, log id: 3b7e38e1
>2018-05-29 12:30:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7f8212e6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:30:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7f8212e6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:30:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [7f8212e6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:30:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [37c95b31] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:30:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [37c95b31] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:30:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [37c95b31] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:30:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [37c95b31] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:30:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [37c95b31] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2fc8d05c
>2018-05-29 12:30:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [37c95b31] FINISH, SetVdsStatusVDSCommand, log id: 2fc8d05c
>2018-05-29 12:30:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [37c95b31] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7fcb2b1a
>2018-05-29 12:30:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [37c95b31] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7fcb2b1a
>2018-05-29 12:30:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [37c95b31] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:30:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [37c95b31] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:30:00,661-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3d1d6f64] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: f4d157e
>2018-05-29 12:30:00,662-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3d1d6f64] FINISH, GetHardwareInfoVDSCommand, log id: f4d157e
>2018-05-29 12:30:00,675-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [57d1258c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:30:00,684-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [73553c34] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:30:00,689-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [25b18251] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:30:00,700-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [25b18251] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3362dc97
>2018-05-29 12:30:01,079-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [25b18251] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:30:01,079-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [25b18251] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3362dc97
>2018-05-29 12:30:01,083-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [25b18251] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:30:01,084-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [25b18251] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:30:01,087-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [25b18251] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:30:01,088-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [25b18251] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:30:01,092-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [25b18251] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:30:01,093-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [25b18251] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:30:01,096-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [25b18251] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:30:01,098-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [25b18251] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 25b18251, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:30:01,105-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: ffee3a1
>2018-05-29 12:30:01,105-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:30:01,202-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:30:01,202-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:30:01,202-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:30:01,202-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:30:01,202-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: ffee3a1
>2018-05-29 12:30:01,202-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:30:01,216-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [1af099a5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:30:01,217-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1af099a5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 10cb0179
>2018-05-29 12:30:01,219-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1af099a5] FINISH, SetVdsStatusVDSCommand, log id: 10cb0179
>2018-05-29 12:30:01,251-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1af099a5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1af099a5, Job ID: 5116faa6-078e-49d2-98f1-dfefee8720c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:30:01,255-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1af099a5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 73553c34, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:30:01,257-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1af099a5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 73553c34, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:30:01,268-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [283089cb] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:30:01,269-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [283089cb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:30:02,364-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [67cc8a2d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 8ba1d92
>2018-05-29 12:30:02,365-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [67cc8a2d] FINISH, GetHardwareInfoVDSCommand, log id: 8ba1d92
>2018-05-29 12:30:02,378-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [714649c0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:30:02,383-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [3712739a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:30:02,388-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [69be7ac4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:30:02,394-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [69be7ac4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7693d110
>2018-05-29 12:30:02,775-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69be7ac4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:30:02,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [69be7ac4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7693d110
>2018-05-29 12:30:02,779-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69be7ac4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:30:02,780-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [69be7ac4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:30:02,783-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69be7ac4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:30:02,784-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [69be7ac4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:30:02,788-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69be7ac4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:30:02,789-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [69be7ac4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:30:02,792-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [69be7ac4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:30:02,795-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69be7ac4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 69be7ac4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:30:02,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5e8962e7
>2018-05-29 12:30:02,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:30:02,898-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:30:02,898-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:30:02,898-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:30:02,898-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:30:02,898-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5e8962e7
>2018-05-29 12:30:02,898-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:30:02,911-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [347de3f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:30:02,912-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [347de3f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17b17efc
>2018-05-29 12:30:02,914-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [347de3f] FINISH, SetVdsStatusVDSCommand, log id: 17b17efc
>2018-05-29 12:30:02,920-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [347de3f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 347de3f, Job ID: 0f86cc41-7561-42d3-aa98-a3e4a7b87f99, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:30:02,924-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [347de3f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3712739a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:30:02,927-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [347de3f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3712739a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:30:02,937-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [1761d0a6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:30:02,938-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1761d0a6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:35:00,016-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [2bfbdc3] Autorecovering 2 hosts
>2018-05-29 12:35:00,016-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [2bfbdc3] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:35:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [49774037] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:35:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [49774037] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:35:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [49774037] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:35:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [49774037] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:35:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [49774037] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50541ebb
>2018-05-29 12:35:00,082-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [49774037] FINISH, SetVdsStatusVDSCommand, log id: 50541ebb
>2018-05-29 12:35:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [49774037] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:35:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [49774037] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:35:00,084-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [49774037] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:35:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5440a16c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:35:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5440a16c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:35:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5440a16c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:35:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5440a16c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:35:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5440a16c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5150af53
>2018-05-29 12:35:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5440a16c] FINISH, SetVdsStatusVDSCommand, log id: 5150af53
>2018-05-29 12:35:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5440a16c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1bbf1298
>2018-05-29 12:35:00,105-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5440a16c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1bbf1298
>2018-05-29 12:35:00,105-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5440a16c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:35:00,106-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5440a16c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:35:00,552-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [561faec1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 24a1143d
>2018-05-29 12:35:00,554-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [561faec1] FINISH, GetHardwareInfoVDSCommand, log id: 24a1143d
>2018-05-29 12:35:00,566-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [56033040] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:35:00,574-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [47828da9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:35:00,579-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [6956d1f3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:35:00,590-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [6956d1f3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 84a5f10
>2018-05-29 12:35:00,974-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6956d1f3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:35:00,974-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [6956d1f3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 84a5f10
>2018-05-29 12:35:00,978-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6956d1f3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:35:00,979-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [6956d1f3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:35:00,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6956d1f3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:35:00,983-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [6956d1f3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:35:00,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6956d1f3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:35:00,988-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [6956d1f3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:35:00,991-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [6956d1f3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:35:00,993-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6956d1f3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6956d1f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:35:01,000-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7793e879
>2018-05-29 12:35:01,000-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:35:01,100-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:35:01,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:35:01,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:35:01,100-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:35:01,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7793e879
>2018-05-29 12:35:01,100-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:35:01,113-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [43dcac6e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:35:01,115-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [43dcac6e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4e16a1fb
>2018-05-29 12:35:01,117-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [43dcac6e] FINISH, SetVdsStatusVDSCommand, log id: 4e16a1fb
>2018-05-29 12:35:01,149-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43dcac6e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 43dcac6e, Job ID: c138ca4b-e69c-4914-8e69-2a5c29c43493, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:35:01,154-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43dcac6e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 47828da9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:35:01,156-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43dcac6e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 47828da9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:35:01,167-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6b072dd0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:35:01,168-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6b072dd0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:35:02,004-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [d17218f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: bec6f00
>2018-05-29 12:35:02,005-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [d17218f] FINISH, GetHardwareInfoVDSCommand, log id: bec6f00
>2018-05-29 12:35:02,018-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [7b9c0dbc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:35:02,026-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [483d03e8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:35:02,031-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [c14dfd7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:35:02,042-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [c14dfd7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 723975fb
>2018-05-29 12:35:03,414-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c14dfd7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:35:03,414-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [c14dfd7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 723975fb
>2018-05-29 12:35:03,418-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c14dfd7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:35:03,419-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [c14dfd7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:35:03,423-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c14dfd7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:35:03,424-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [c14dfd7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:35:03,427-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c14dfd7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:35:03,428-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [c14dfd7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:35:03,431-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [c14dfd7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:35:03,434-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c14dfd7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: c14dfd7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:35:03,437-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3962c88f
>2018-05-29 12:35:03,437-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:35:03,531-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:35:03,531-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:35:03,531-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:35:03,531-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:35:03,531-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3962c88f
>2018-05-29 12:35:03,531-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:35:03,545-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [735add89] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:35:03,546-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [735add89] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19b9a236
>2018-05-29 12:35:03,548-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [735add89] FINISH, SetVdsStatusVDSCommand, log id: 19b9a236
>2018-05-29 12:35:03,554-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [735add89] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 735add89, Job ID: 063279ec-1118-4693-b7b2-13855c747244, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:35:03,558-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [735add89] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 483d03e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:35:03,561-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [735add89] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 483d03e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:35:03,571-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [2469b916] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:35:03,572-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2469b916] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:40:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [b264439] Autorecovering 2 hosts
>2018-05-29 12:40:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [b264439] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:40:00,031-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17113a2c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:40:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17113a2c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:40:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17113a2c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:40:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17113a2c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:40:00,040-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [17113a2c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ca9dc4a
>2018-05-29 12:40:00,043-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [17113a2c] FINISH, SetVdsStatusVDSCommand, log id: 6ca9dc4a
>2018-05-29 12:40:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17113a2c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:40:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17113a2c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:40:00,046-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [17113a2c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:40:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [563147d6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:40:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [563147d6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:40:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [563147d6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:40:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [563147d6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:40:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [563147d6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3eb894b5
>2018-05-29 12:40:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [563147d6] FINISH, SetVdsStatusVDSCommand, log id: 3eb894b5
>2018-05-29 12:40:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [563147d6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 70dc8e36
>2018-05-29 12:40:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [563147d6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 70dc8e36
>2018-05-29 12:40:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [563147d6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:40:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [563147d6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:40:01,324-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [32a9491e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3c0ee727
>2018-05-29 12:40:01,326-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [32a9491e] FINISH, GetHardwareInfoVDSCommand, log id: 3c0ee727
>2018-05-29 12:40:01,339-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [3c5f916c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:40:01,347-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [1b2d1d89] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:40:01,353-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6a76aa3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:40:01,359-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6a76aa3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4188d33e
>2018-05-29 12:40:01,728-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a76aa3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:40:01,728-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6a76aa3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4188d33e
>2018-05-29 12:40:01,731-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a76aa3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:40:01,732-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6a76aa3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:40:01,736-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a76aa3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:40:01,737-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6a76aa3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:40:01,740-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a76aa3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:40:01,741-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6a76aa3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:40:01,744-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6a76aa3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:40:01,747-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a76aa3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6a76aa3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:40:01,750-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 45036e64
>2018-05-29 12:40:01,750-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:40:01,792-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [19c0cad2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 67349e7d
>2018-05-29 12:40:01,794-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [19c0cad2] FINISH, GetHardwareInfoVDSCommand, log id: 67349e7d
>2018-05-29 12:40:01,806-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [56ba86d2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:40:01,810-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [3f829868] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:40:01,815-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [18770b6f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:40:01,821-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [18770b6f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 20a1a1a8
>2018-05-29 12:40:01,848-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:40:01,848-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:40:01,848-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:40:01,848-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:40:01,848-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 45036e64
>2018-05-29 12:40:01,848-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:40:01,862-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [3354cef9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:40:01,863-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3354cef9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f9bf088
>2018-05-29 12:40:01,865-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3354cef9] FINISH, SetVdsStatusVDSCommand, log id: 5f9bf088
>2018-05-29 12:40:01,895-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3354cef9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3354cef9, Job ID: 0a588721-10cf-4c4c-82bc-3fc15596e981, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:40:01,899-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3354cef9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1b2d1d89, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:40:01,901-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3354cef9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1b2d1d89, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:40:01,912-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [14601506] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:40:01,913-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [14601506] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:40:02,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18770b6f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:40:02,198-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [18770b6f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 20a1a1a8
>2018-05-29 12:40:02,202-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18770b6f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:40:02,203-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [18770b6f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:40:02,207-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18770b6f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:40:02,207-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [18770b6f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:40:02,211-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18770b6f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:40:02,212-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [18770b6f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:40:02,215-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [18770b6f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:40:02,217-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18770b6f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 18770b6f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:40:02,220-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6b0706b7
>2018-05-29 12:40:02,220-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:40:02,315-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:40:02,315-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:40:02,315-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:40:02,315-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:40:02,315-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6b0706b7
>2018-05-29 12:40:02,315-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:40:02,329-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1b1ec9ae] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:40:02,330-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1b1ec9ae] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3797622e
>2018-05-29 12:40:02,332-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1b1ec9ae] FINISH, SetVdsStatusVDSCommand, log id: 3797622e
>2018-05-29 12:40:02,338-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1b1ec9ae] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1b1ec9ae, Job ID: 4a47734c-f9ac-4b4b-be8a-e431f100364f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:40:02,342-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1b1ec9ae] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3f829868, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:40:02,344-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1b1ec9ae] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3f829868, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:40:02,355-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [6cd638a7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:40:02,356-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [6cd638a7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [57e2197f] Autorecovering 2 hosts
>2018-05-29 12:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [57e2197f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:45:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1a6eb640] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:45:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1a6eb640] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:45:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1a6eb640] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:45:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1a6eb640] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:45:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1a6eb640] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d8d59e
>2018-05-29 12:45:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1a6eb640] FINISH, SetVdsStatusVDSCommand, log id: 2d8d59e
>2018-05-29 12:45:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1a6eb640] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:45:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1a6eb640] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:45:00,073-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [1a6eb640] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:45:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [558cf7b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:45:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [558cf7b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:45:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [558cf7b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:45:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [558cf7b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:45:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [558cf7b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6e5053c5
>2018-05-29 12:45:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [558cf7b] FINISH, SetVdsStatusVDSCommand, log id: 6e5053c5
>2018-05-29 12:45:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [558cf7b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1782dd9d
>2018-05-29 12:45:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [558cf7b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1782dd9d
>2018-05-29 12:45:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [558cf7b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:45:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [558cf7b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:45:02,593-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [121c4e1c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3fe7c78c
>2018-05-29 12:45:02,594-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [121c4e1c] FINISH, GetHardwareInfoVDSCommand, log id: 3fe7c78c
>2018-05-29 12:45:02,608-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [777e7dd5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:45:02,616-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [1d1ea1fa] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:45:02,621-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [693315a9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:45:02,634-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [693315a9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 16ed3f2d
>2018-05-29 12:45:02,948-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [27606eab] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3c09caa4
>2018-05-29 12:45:02,949-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [27606eab] FINISH, GetHardwareInfoVDSCommand, log id: 3c09caa4
>2018-05-29 12:45:02,962-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [a6fab29] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:45:02,966-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6d486be6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:45:02,971-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [ffd623f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:45:02,977-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [ffd623f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 46bd5f6b
>2018-05-29 12:45:02,998-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [693315a9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:45:02,998-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [693315a9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 16ed3f2d
>2018-05-29 12:45:03,002-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [693315a9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:45:03,003-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [693315a9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:45:03,007-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [693315a9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:45:03,007-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [693315a9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:45:03,011-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [693315a9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:45:03,012-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [693315a9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:45:03,014-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [693315a9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:45:03,017-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [693315a9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 693315a9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:45:03,025-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1cc3fb89
>2018-05-29 12:45:03,025-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:45:03,122-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:45:03,122-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:45:03,122-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:45:03,122-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:45:03,122-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1cc3fb89
>2018-05-29 12:45:03,122-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:45:03,136-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [5d97cd2a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:45:03,137-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5d97cd2a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 51088bef
>2018-05-29 12:45:03,139-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5d97cd2a] FINISH, SetVdsStatusVDSCommand, log id: 51088bef
>2018-05-29 12:45:03,172-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5d97cd2a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5d97cd2a, Job ID: 2816a803-7a75-435f-9f0a-1f94a323ce75, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:45:03,177-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5d97cd2a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d1ea1fa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:45:03,179-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5d97cd2a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d1ea1fa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:45:03,189-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [778ace6e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:45:03,190-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [778ace6e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:45:03,350-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ffd623f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:45:03,350-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [ffd623f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 46bd5f6b
>2018-05-29 12:45:03,354-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ffd623f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:45:03,355-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [ffd623f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:45:03,358-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ffd623f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:45:03,359-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [ffd623f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:45:03,363-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ffd623f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:45:03,364-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [ffd623f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:45:03,366-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [ffd623f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:45:03,369-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ffd623f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: ffd623f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:45:03,372-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1767b82c
>2018-05-29 12:45:03,372-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:45:03,473-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:45:03,473-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:45:03,473-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:45:03,473-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:45:03,473-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1767b82c
>2018-05-29 12:45:03,473-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:45:03,486-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [555ac4b9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:45:03,487-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [555ac4b9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7d3b06b6
>2018-05-29 12:45:03,490-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [555ac4b9] FINISH, SetVdsStatusVDSCommand, log id: 7d3b06b6
>2018-05-29 12:45:03,495-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [555ac4b9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 555ac4b9, Job ID: 575ee1f8-9a96-47f2-bc56-ba55d0d40b81, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:45:03,500-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [555ac4b9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6d486be6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:45:03,502-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [555ac4b9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6d486be6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:45:03,513-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [34740f34] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:45:03,514-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [34740f34] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1ba04a93] Autorecovering 2 hosts
>2018-05-29 12:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1ba04a93] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:50:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a2cd813] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:50:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a2cd813] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:50:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a2cd813] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:50:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a2cd813] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:50:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2a2cd813] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5449997b
>2018-05-29 12:50:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2a2cd813] FINISH, SetVdsStatusVDSCommand, log id: 5449997b
>2018-05-29 12:50:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a2cd813] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:50:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a2cd813] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:50:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2a2cd813] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:50:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2f15c8da] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:50:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2f15c8da] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:50:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2f15c8da] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:50:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2f15c8da] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:50:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2f15c8da] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46aa70b1
>2018-05-29 12:50:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2f15c8da] FINISH, SetVdsStatusVDSCommand, log id: 46aa70b1
>2018-05-29 12:50:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2f15c8da] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1212c07
>2018-05-29 12:50:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2f15c8da] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1212c07
>2018-05-29 12:50:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2f15c8da] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:50:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2f15c8da] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:50:00,906-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [59babb5d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 65fc67df
>2018-05-29 12:50:00,908-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [59babb5d] FINISH, GetHardwareInfoVDSCommand, log id: 65fc67df
>2018-05-29 12:50:00,921-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [245875cd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:50:00,930-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [189942a5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:50:00,935-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [192aebf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:50:00,947-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [192aebf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3db81714
>2018-05-29 12:50:01,103-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6b931838] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4c86ebf4
>2018-05-29 12:50:01,105-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6b931838] FINISH, GetHardwareInfoVDSCommand, log id: 4c86ebf4
>2018-05-29 12:50:01,117-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [57c57f31] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:50:01,121-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [602d77b4] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:50:01,126-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [234b7ade] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:50:01,132-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [234b7ade] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7d0e106f
>2018-05-29 12:50:01,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [192aebf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:50:01,327-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [192aebf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3db81714
>2018-05-29 12:50:01,331-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [192aebf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:50:01,332-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [192aebf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:50:01,335-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [192aebf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:50:01,336-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [192aebf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:50:01,339-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [192aebf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:50:01,340-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [192aebf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:50:01,343-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [192aebf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:50:01,346-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [192aebf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 192aebf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:50:01,352-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1b14e3d
>2018-05-29 12:50:01,352-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:50:01,450-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:50:01,450-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:50:01,450-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:50:01,450-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:50:01,450-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1b14e3d
>2018-05-29 12:50:01,450-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:50:01,463-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [1e8d9a0b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:50:01,464-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1e8d9a0b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46e68d54
>2018-05-29 12:50:01,466-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1e8d9a0b] FINISH, SetVdsStatusVDSCommand, log id: 46e68d54
>2018-05-29 12:50:01,498-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1e8d9a0b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1e8d9a0b, Job ID: 41c2339b-fc93-4d62-863f-d856008f6504, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:50:01,503-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1e8d9a0b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 189942a5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:50:01,505-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1e8d9a0b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 189942a5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:50:01,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [234b7ade] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:50:01,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [234b7ade] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7d0e106f
>2018-05-29 12:50:01,515-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [2bc57482] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:50:01,516-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [2bc57482] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:50:01,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [234b7ade] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:50:01,517-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [234b7ade] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:50:01,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [234b7ade] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:50:01,522-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [234b7ade] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:50:01,526-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [234b7ade] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:50:01,526-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [234b7ade] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:50:01,529-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [234b7ade] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:50:01,532-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [234b7ade] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 234b7ade, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:50:01,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5fbcbf5b
>2018-05-29 12:50:01,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:50:01,632-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:50:01,632-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:50:01,632-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:50:01,632-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:50:01,632-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5fbcbf5b
>2018-05-29 12:50:01,632-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:50:01,645-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [3702e7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:50:01,646-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3702e7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1922315d
>2018-05-29 12:50:01,648-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3702e7] FINISH, SetVdsStatusVDSCommand, log id: 1922315d
>2018-05-29 12:50:01,654-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3702e7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3702e7, Job ID: d2eec3d7-2e3c-4ac6-b11a-7ef4f3715a83, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:50:01,658-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3702e7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 602d77b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:50:01,660-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3702e7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 602d77b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:50:01,671-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7b924a89] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:50:01,672-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7b924a89] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:54:43,016-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler2) [5519621c] Attempting to update VMs/Templates Ovf.
>2018-05-29 12:54:43,019-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 12:54:43,019-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 12:54:43,020-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:54:43,024-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 12:54:43,028-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 12:54:43,028-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 12:54:43,028-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 12:54:43,028-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 12:54:43,029-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 12:54:43,029-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 12:55:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7a28deaf] Autorecovering 2 hosts
>2018-05-29 12:55:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7a28deaf] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:55:00,031-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1ba2bd76] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:55:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1ba2bd76] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:55:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1ba2bd76] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:55:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1ba2bd76] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:55:00,039-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1ba2bd76] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 585eb811
>2018-05-29 12:55:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1ba2bd76] FINISH, SetVdsStatusVDSCommand, log id: 585eb811
>2018-05-29 12:55:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1ba2bd76] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:55:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1ba2bd76] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 12:55:00,044-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1ba2bd76] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:55:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [f6f4dcc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:55:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [f6f4dcc] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 12:55:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [f6f4dcc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:55:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [f6f4dcc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:55:00,057-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [f6f4dcc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 635a06df
>2018-05-29 12:55:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [f6f4dcc] FINISH, SetVdsStatusVDSCommand, log id: 635a06df
>2018-05-29 12:55:00,062-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [f6f4dcc] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 372d807c
>2018-05-29 12:55:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [f6f4dcc] FINISH, SetHaMaintenanceModeVDSCommand, log id: 372d807c
>2018-05-29 12:55:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [f6f4dcc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 12:55:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [f6f4dcc] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 12:55:02,154-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6b6c5124] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 633eb109
>2018-05-29 12:55:02,156-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6b6c5124] FINISH, GetHardwareInfoVDSCommand, log id: 633eb109
>2018-05-29 12:55:02,169-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [3f4349ab] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:55:02,177-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [789b7002] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:55:02,182-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4b5a11e2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:55:02,194-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4b5a11e2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 523aaa1c
>2018-05-29 12:55:02,256-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [24e5a684] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6b11071d
>2018-05-29 12:55:02,257-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [24e5a684] FINISH, GetHardwareInfoVDSCommand, log id: 6b11071d
>2018-05-29 12:55:02,269-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [68f4b6aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:55:02,274-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1ae4d5ae] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:55:02,279-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [419e0371] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 12:55:02,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [419e0371] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1dad0c4f
>2018-05-29 12:55:02,671-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e0371] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:55:02,671-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [419e0371] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1dad0c4f
>2018-05-29 12:55:02,675-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e0371] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:55:02,675-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [419e0371] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:55:02,679-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e0371] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:55:02,679-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [419e0371] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:55:02,683-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e0371] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:55:02,684-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [419e0371] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:55:02,687-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [419e0371] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:55:02,689-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e0371] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 419e0371, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:55:02,697-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 22373935
>2018-05-29 12:55:02,697-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:55:02,797-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:55:02,797-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:55:02,797-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:55:02,797-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:55:02,797-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 22373935
>2018-05-29 12:55:02,797-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:55:02,810-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [251d481e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:55:02,811-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [251d481e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 804e2cd
>2018-05-29 12:55:02,814-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [251d481e] FINISH, SetVdsStatusVDSCommand, log id: 804e2cd
>2018-05-29 12:55:02,848-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [251d481e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 251d481e, Job ID: 42f9ca34-59cf-4bed-b754-51652223c5b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:55:02,853-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [251d481e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1ae4d5ae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:55:02,855-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [251d481e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1ae4d5ae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:55:02,866-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [699965d3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 12:55:02,867-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [699965d3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 12:55:03,559-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4b5a11e2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 12:55:03,559-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4b5a11e2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 523aaa1c
>2018-05-29 12:55:03,563-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4b5a11e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:55:03,563-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4b5a11e2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:55:03,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4b5a11e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:55:03,568-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4b5a11e2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:55:03,571-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4b5a11e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 12:55:03,572-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4b5a11e2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 12:55:03,575-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4b5a11e2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 12:55:03,578-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4b5a11e2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4b5a11e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 12:55:03,582-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6a67fb81
>2018-05-29 12:55:03,582-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 12:55:03,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:55:03,680-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 12:55:03,680-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 12:55:03,680-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 12:55:03,680-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6a67fb81
>2018-05-29 12:55:03,680-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 12:55:03,692-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4911c361] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:55:03,694-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4911c361] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43d32ff0
>2018-05-29 12:55:03,696-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4911c361] FINISH, SetVdsStatusVDSCommand, log id: 43d32ff0
>2018-05-29 12:55:03,701-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4911c361] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4911c361, Job ID: f1c376a7-273f-4ff4-9f64-e11770d26782, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 12:55:03,706-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4911c361] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 789b7002, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 12:55:03,708-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4911c361] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 789b7002, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 12:55:03,718-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [70c0cff8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 12:55:03,719-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [70c0cff8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6e6ef36d] Autorecovering 2 hosts
>2018-05-29 13:00:00,011-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6e6ef36d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:00:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [110a0f3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [110a0f3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [110a0f3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [110a0f3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:00:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [110a0f3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e911d2
>2018-05-29 13:00:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [110a0f3] FINISH, SetVdsStatusVDSCommand, log id: 5e911d2
>2018-05-29 13:00:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [110a0f3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:00:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [110a0f3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:00:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [110a0f3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:00:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1d91a023] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1d91a023] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1d91a023] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1d91a023] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:00:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1d91a023] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 178a3c67
>2018-05-29 13:00:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1d91a023] FINISH, SetVdsStatusVDSCommand, log id: 178a3c67
>2018-05-29 13:00:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [1d91a023] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 190a2184
>2018-05-29 13:00:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [1d91a023] FINISH, SetHaMaintenanceModeVDSCommand, log id: 190a2184
>2018-05-29 13:00:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1d91a023] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:00:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1d91a023] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:00:00,490-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [6068bdce] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7ddb1d60
>2018-05-29 13:00:00,492-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [6068bdce] FINISH, GetHardwareInfoVDSCommand, log id: 7ddb1d60
>2018-05-29 13:00:00,505-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [750f8f0f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:00:00,513-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [4544d1d9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:00:00,518-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [316001c4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:00:00,529-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [316001c4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 17866d2d
>2018-05-29 13:00:00,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [316001c4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:00:00,912-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [316001c4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 17866d2d
>2018-05-29 13:00:00,916-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [316001c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:00:00,917-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [316001c4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:00:00,921-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [316001c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:00:00,922-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [316001c4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:00:00,926-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [316001c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:00:00,927-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [316001c4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:00:00,930-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [316001c4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:00:00,934-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [316001c4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 316001c4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:00:00,940-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1dfc1e1b
>2018-05-29 13:00:00,940-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:00:01,037-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:00:01,037-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:00:01,037-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:00:01,037-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:00:01,037-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1dfc1e1b
>2018-05-29 13:00:01,037-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:00:01,051-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [6259e36a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:00:01,052-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6259e36a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7006711b
>2018-05-29 13:00:01,055-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6259e36a] FINISH, SetVdsStatusVDSCommand, log id: 7006711b
>2018-05-29 13:00:01,086-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6259e36a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6259e36a, Job ID: 65c37659-df71-4c10-b64e-782dcc2cd805, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:00:01,091-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6259e36a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4544d1d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:00:01,093-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6259e36a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4544d1d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:00:01,103-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [69b80439] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:00:01,104-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [69b80439] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:00:01,381-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3662aae4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 300967c8
>2018-05-29 13:00:01,383-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3662aae4] FINISH, GetHardwareInfoVDSCommand, log id: 300967c8
>2018-05-29 13:00:01,396-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [275dfcc8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:00:01,401-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1ea93fc8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:00:01,406-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3059539e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:00:01,412-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3059539e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7238a330
>2018-05-29 13:00:01,789-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3059539e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:00:01,789-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3059539e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7238a330
>2018-05-29 13:00:01,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3059539e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:00:01,793-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3059539e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:00:01,797-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3059539e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:00:01,798-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3059539e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:00:01,801-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3059539e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:00:01,802-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3059539e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:00:01,804-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3059539e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:00:01,807-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3059539e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3059539e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:00:01,814-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 8d343e7
>2018-05-29 13:00:01,814-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:00:01,916-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:00:01,916-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:00:01,916-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:00:01,916-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:00:01,916-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 8d343e7
>2018-05-29 13:00:01,916-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:00:01,929-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [4b720646] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:00:01,930-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4b720646] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4f008477
>2018-05-29 13:00:01,932-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4b720646] FINISH, SetVdsStatusVDSCommand, log id: 4f008477
>2018-05-29 13:00:01,965-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4b720646] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4b720646, Job ID: 0da18e1c-4424-4e41-b390-66497acf0513, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:00:01,970-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4b720646] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1ea93fc8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:00:01,972-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4b720646] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1ea93fc8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:00:01,982-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [3ada6569] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:00:01,983-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [3ada6569] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5fd41efb] Autorecovering 2 hosts
>2018-05-29 13:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5fd41efb] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:05:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b299fc9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b299fc9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b299fc9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b299fc9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:05:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7b299fc9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 727ef815
>2018-05-29 13:05:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7b299fc9] FINISH, SetVdsStatusVDSCommand, log id: 727ef815
>2018-05-29 13:05:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b299fc9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:05:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b299fc9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:05:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7b299fc9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:05:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b2b119] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b2b119] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b2b119] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b2b119] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:05:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [54b2b119] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 70bbed18
>2018-05-29 13:05:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [54b2b119] FINISH, SetVdsStatusVDSCommand, log id: 70bbed18
>2018-05-29 13:05:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [54b2b119] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 23df1aed
>2018-05-29 13:05:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [54b2b119] FINISH, SetHaMaintenanceModeVDSCommand, log id: 23df1aed
>2018-05-29 13:05:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b2b119] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:05:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b2b119] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:05:01,692-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [403d5d1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2aa2ac18
>2018-05-29 13:05:01,694-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [403d5d1] FINISH, GetHardwareInfoVDSCommand, log id: 2aa2ac18
>2018-05-29 13:05:01,707-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [325d4d02] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:05:01,715-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [17f700d2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:05:01,720-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [7219cca] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:05:01,732-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [7219cca] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 667e6507
>2018-05-29 13:05:02,118-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7219cca] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:05:02,118-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [7219cca] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 667e6507
>2018-05-29 13:05:02,122-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7219cca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:05:02,122-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [7219cca] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:05:02,127-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7219cca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:05:02,127-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [7219cca] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:05:02,131-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7219cca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:05:02,132-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [7219cca] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:05:02,134-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [7219cca] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:05:02,137-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7219cca] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7219cca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:05:02,144-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2c28e27f
>2018-05-29 13:05:02,144-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:05:02,244-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:05:02,244-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:05:02,244-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:05:02,244-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:05:02,244-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2c28e27f
>2018-05-29 13:05:02,244-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:05:02,258-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [45bf7a1e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:05:02,259-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [45bf7a1e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4381e6a0
>2018-05-29 13:05:02,261-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [45bf7a1e] FINISH, SetVdsStatusVDSCommand, log id: 4381e6a0
>2018-05-29 13:05:02,295-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [45bf7a1e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 45bf7a1e, Job ID: 7a34a253-4105-4ffb-8e71-189c7db7c110, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:05:02,300-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [45bf7a1e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17f700d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:05:02,302-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [45bf7a1e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17f700d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:05:02,313-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [e8bea6a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:05:02,314-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [e8bea6a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:05:02,710-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [10ae557e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 14f521fd
>2018-05-29 13:05:02,712-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [10ae557e] FINISH, GetHardwareInfoVDSCommand, log id: 14f521fd
>2018-05-29 13:05:02,724-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [4042d581] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:05:02,729-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [64f725c1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:05:02,734-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5fcbfea0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:05:02,742-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5fcbfea0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 449b1053
>2018-05-29 13:05:05,062-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5fcbfea0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:05:05,062-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5fcbfea0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 449b1053
>2018-05-29 13:05:05,066-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5fcbfea0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:05:05,067-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5fcbfea0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:05:05,070-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5fcbfea0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:05:05,071-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5fcbfea0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:05:05,075-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5fcbfea0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:05:05,076-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5fcbfea0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:05:05,080-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5fcbfea0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:05:05,082-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5fcbfea0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5fcbfea0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:05:05,085-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 25fbe065
>2018-05-29 13:05:05,085-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:05:05,184-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:05:05,184-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:05:05,184-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:05:05,184-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:05:05,184-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 25fbe065
>2018-05-29 13:05:05,184-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:05:05,197-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [38032122] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:05:05,198-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [38032122] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31ea8f0f
>2018-05-29 13:05:05,200-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [38032122] FINISH, SetVdsStatusVDSCommand, log id: 31ea8f0f
>2018-05-29 13:05:05,206-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [38032122] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 38032122, Job ID: a43a3c62-ad6a-43cf-b149-30355203f773, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:05:05,210-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [38032122] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 64f725c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:05:05,213-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [38032122] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 64f725c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:05:05,224-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [316ce5cd] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:05:05,225-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [316ce5cd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3703382a] Autorecovering 2 hosts
>2018-05-29 13:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3703382a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:10:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4a99d26c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:10:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4a99d26c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:10:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4a99d26c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:10:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4a99d26c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:10:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4a99d26c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26857632
>2018-05-29 13:10:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4a99d26c] FINISH, SetVdsStatusVDSCommand, log id: 26857632
>2018-05-29 13:10:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4a99d26c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:10:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4a99d26c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:10:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4a99d26c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:10:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8e1e0f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:10:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8e1e0f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:10:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8e1e0f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:10:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8e1e0f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:10:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3a8e1e0f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a8c021d
>2018-05-29 13:10:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3a8e1e0f] FINISH, SetVdsStatusVDSCommand, log id: a8c021d
>2018-05-29 13:10:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3a8e1e0f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3eaab737
>2018-05-29 13:10:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3a8e1e0f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3eaab737
>2018-05-29 13:10:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8e1e0f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:10:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8e1e0f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:10:02,959-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [dc90603] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5cca8bd6
>2018-05-29 13:10:02,961-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [dc90603] FINISH, GetHardwareInfoVDSCommand, log id: 5cca8bd6
>2018-05-29 13:10:02,975-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [343e0995] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:10:02,983-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1e4f73b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:10:02,989-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [11ee8734] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:10:02,999-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1e0e8d42] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 590002e7
>2018-05-29 13:10:03,000-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1e0e8d42] FINISH, GetHardwareInfoVDSCommand, log id: 590002e7
>2018-05-29 13:10:03,015-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4707db9b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:10:03,024-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [6164ba54] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:10:03,029-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [479798f0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:10:03,041-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [479798f0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 73e2445e
>2018-05-29 13:10:03,048-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [11ee8734] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4d4c46e7
>2018-05-29 13:10:03,414-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [479798f0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:10:03,414-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [479798f0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 73e2445e
>2018-05-29 13:10:03,419-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [479798f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:10:03,420-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [479798f0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:10:03,424-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [479798f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:10:03,425-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [479798f0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:10:03,429-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [11ee8734] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:10:03,429-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [11ee8734] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4d4c46e7
>2018-05-29 13:10:03,433-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [11ee8734] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:10:03,434-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [11ee8734] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:10:03,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [479798f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:10:03,437-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [479798f0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:10:03,439-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [11ee8734] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:10:03,439-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [11ee8734] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:10:03,440-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [479798f0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:10:03,443-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [479798f0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 479798f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:10:03,444-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [11ee8734] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:10:03,445-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [11ee8734] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:10:03,447-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [11ee8734] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:10:03,450-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [11ee8734] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 11ee8734, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:10:03,450-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 457491e0
>2018-05-29 13:10:03,450-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:10:03,545-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:10:03,545-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:10:03,545-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:10:03,545-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:10:03,545-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 457491e0
>2018-05-29 13:10:03,546-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:10:03,551-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 56ac405b
>2018-05-29 13:10:03,551-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:10:03,558-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [45d91327] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:10:03,560-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [45d91327] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 653f413c
>2018-05-29 13:10:03,562-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [45d91327] FINISH, SetVdsStatusVDSCommand, log id: 653f413c
>2018-05-29 13:10:03,570-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [45d91327] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 45d91327, Job ID: dd7c2a65-38f4-47ee-8c5c-c57bb2eeab08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:10:03,574-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [45d91327] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6164ba54, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:10:03,576-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [45d91327] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6164ba54, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:10:03,587-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [2a5c0fa3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:10:03,588-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [2a5c0fa3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:10:03,651-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:10:03,651-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:10:03,651-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:10:03,651-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:10:03,651-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 56ac405b
>2018-05-29 13:10:03,651-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:10:03,664-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [ad9c703] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:10:03,665-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [ad9c703] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b102443
>2018-05-29 13:10:03,667-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [ad9c703] FINISH, SetVdsStatusVDSCommand, log id: 1b102443
>2018-05-29 13:10:03,672-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [ad9c703] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: ad9c703, Job ID: 5a7a3034-b0c0-4637-80e9-456671064c38, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:10:03,677-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [ad9c703] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1e4f73b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:10:03,679-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [ad9c703] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1e4f73b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:10:03,689-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [613a8c3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:10:03,690-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [613a8c3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:15:00,011-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1ff9aa2d] Autorecovering 2 hosts
>2018-05-29 13:15:00,011-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1ff9aa2d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:15:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [17cf8043] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [17cf8043] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [17cf8043] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [17cf8043] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:15:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [17cf8043] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3cad7c56
>2018-05-29 13:15:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [17cf8043] FINISH, SetVdsStatusVDSCommand, log id: 3cad7c56
>2018-05-29 13:15:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [17cf8043] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:15:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [17cf8043] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:15:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [17cf8043] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:15:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7ef27c21] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:15:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7ef27c21] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:15:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7ef27c21] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:15:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7ef27c21] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:15:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7ef27c21] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7271a2ea
>2018-05-29 13:15:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7ef27c21] FINISH, SetVdsStatusVDSCommand, log id: 7271a2ea
>2018-05-29 13:15:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [7ef27c21] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3a53653
>2018-05-29 13:15:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [7ef27c21] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3a53653
>2018-05-29 13:15:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7ef27c21] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:15:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7ef27c21] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:15:01,325-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [77319510] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6865854f
>2018-05-29 13:15:01,327-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [77319510] FINISH, GetHardwareInfoVDSCommand, log id: 6865854f
>2018-05-29 13:15:01,339-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [7eca4b80] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:15:01,348-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [5afb0565] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:15:01,353-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [8f534c6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:15:01,360-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [311d58fc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1b407d96
>2018-05-29 13:15:01,360-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [8f534c6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6a3a60fc
>2018-05-29 13:15:01,362-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [311d58fc] FINISH, GetHardwareInfoVDSCommand, log id: 1b407d96
>2018-05-29 13:15:01,374-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [5fb658a4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:15:01,378-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [660b2bca] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:15:01,383-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5e2427f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:15:01,389-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5e2427f8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 40d5bf74
>2018-05-29 13:15:01,732-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [8f534c6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:15:01,732-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [8f534c6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6a3a60fc
>2018-05-29 13:15:01,736-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [8f534c6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:15:01,737-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [8f534c6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:15:01,741-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [8f534c6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:15:01,741-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [8f534c6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:15:01,745-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [8f534c6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:15:01,746-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [8f534c6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:15:01,750-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [8f534c6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:15:01,752-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [8f534c6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 8f534c6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:15:01,755-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 13e0ea10
>2018-05-29 13:15:01,755-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:15:01,765-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e2427f8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:15:01,765-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5e2427f8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 40d5bf74
>2018-05-29 13:15:01,769-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e2427f8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:15:01,769-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5e2427f8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:15:01,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e2427f8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:15:01,774-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5e2427f8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:15:01,778-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e2427f8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:15:01,778-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5e2427f8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:15:01,781-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5e2427f8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:15:01,784-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e2427f8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5e2427f8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:15:01,857-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:15:01,857-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:15:01,857-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:15:01,857-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:15:01,858-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 13e0ea10
>2018-05-29 13:15:01,858-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:15:01,859-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 49df0673
>2018-05-29 13:15:01,859-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:15:01,870-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [7085e96f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:15:01,872-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7085e96f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76e2131b
>2018-05-29 13:15:01,874-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7085e96f] FINISH, SetVdsStatusVDSCommand, log id: 76e2131b
>2018-05-29 13:15:01,905-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7085e96f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7085e96f, Job ID: 3fe0a810-e4eb-4690-bf29-9e12b32e9962, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:15:01,909-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7085e96f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5afb0565, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:15:01,912-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7085e96f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5afb0565, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:15:01,923-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [33423315] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:15:01,924-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [33423315] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:15:01,961-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:15:01,961-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:15:01,961-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:15:01,961-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:15:01,961-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 49df0673
>2018-05-29 13:15:01,961-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:15:01,974-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [d174964] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:15:01,976-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [d174964] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 10a301be
>2018-05-29 13:15:01,978-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [d174964] FINISH, SetVdsStatusVDSCommand, log id: 10a301be
>2018-05-29 13:15:01,983-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d174964] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: d174964, Job ID: 264d8f4c-710e-4de2-8be3-a2e0211f1929, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:15:01,988-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d174964] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 660b2bca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:15:01,990-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d174964] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 660b2bca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:15:02,000-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [50de081c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:15:02,001-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [50de081c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:20:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [35e2ea4a] Autorecovering 2 hosts
>2018-05-29 13:20:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [35e2ea4a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:20:00,033-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6a6f654d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:20:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6a6f654d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:20:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6a6f654d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:20:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6a6f654d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:20:00,041-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6a6f654d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4a48cc90
>2018-05-29 13:20:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6a6f654d] FINISH, SetVdsStatusVDSCommand, log id: 4a48cc90
>2018-05-29 13:20:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6a6f654d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:20:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6a6f654d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:20:00,046-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6a6f654d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:20:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1336f75b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:20:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1336f75b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:20:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1336f75b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:20:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1336f75b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:20:00,060-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1336f75b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 879a272
>2018-05-29 13:20:00,062-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1336f75b] FINISH, SetVdsStatusVDSCommand, log id: 879a272
>2018-05-29 13:20:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [1336f75b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6bb9c6de
>2018-05-29 13:20:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [1336f75b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6bb9c6de
>2018-05-29 13:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1336f75b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:20:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1336f75b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:20:02,599-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [258a57f7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7d3efbcb
>2018-05-29 13:20:02,601-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [258a57f7] FINISH, GetHardwareInfoVDSCommand, log id: 7d3efbcb
>2018-05-29 13:20:02,614-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [52ee24f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:20:02,622-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [56ed6b5a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:20:02,628-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [363f384e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:20:02,637-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4c3cd69c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 40c9be86
>2018-05-29 13:20:02,639-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4c3cd69c] FINISH, GetHardwareInfoVDSCommand, log id: 40c9be86
>2018-05-29 13:20:02,641-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [363f384e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c184893
>2018-05-29 13:20:02,651-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [447830c3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:20:02,655-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [92a35c7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:20:02,661-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [17f5f7fc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:20:02,673-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [17f5f7fc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 59763160
>2018-05-29 13:20:03,019-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [363f384e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:20:03,019-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [363f384e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c184893
>2018-05-29 13:20:03,023-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [363f384e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:20:03,024-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [363f384e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:20:03,028-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [363f384e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:20:03,028-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [363f384e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:20:03,036-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [363f384e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:20:03,037-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [363f384e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:20:03,040-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [363f384e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:20:03,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [17f5f7fc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:20:03,042-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [17f5f7fc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 59763160
>2018-05-29 13:20:03,043-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [363f384e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 363f384e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:20:03,046-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [17f5f7fc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:20:03,046-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [17f5f7fc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:20:03,050-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5e72834
>2018-05-29 13:20:03,050-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:20:03,050-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [17f5f7fc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:20:03,051-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [17f5f7fc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:20:03,055-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [17f5f7fc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:20:03,056-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [17f5f7fc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:20:03,058-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [17f5f7fc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:20:03,061-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [17f5f7fc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 17f5f7fc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:20:03,148-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:20:03,148-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:20:03,148-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:20:03,148-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:20:03,148-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5e72834
>2018-05-29 13:20:03,148-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:20:03,154-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: f311b54
>2018-05-29 13:20:03,154-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:20:03,162-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [e1b50ca] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:20:03,163-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [e1b50ca] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5febb576
>2018-05-29 13:20:03,166-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [e1b50ca] FINISH, SetVdsStatusVDSCommand, log id: 5febb576
>2018-05-29 13:20:03,174-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e1b50ca] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: e1b50ca, Job ID: 8a220d70-03e1-42c1-98a2-b014253982ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:20:03,179-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e1b50ca] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 56ed6b5a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:20:03,181-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e1b50ca] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 56ed6b5a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:20:03,192-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [182bf31a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:20:03,193-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [182bf31a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:20:03,251-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:20:03,251-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:20:03,251-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:20:03,251-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:20:03,251-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: f311b54
>2018-05-29 13:20:03,251-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:20:03,264-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [6c896735] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:20:03,265-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6c896735] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7391fd2f
>2018-05-29 13:20:03,267-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6c896735] FINISH, SetVdsStatusVDSCommand, log id: 7391fd2f
>2018-05-29 13:20:03,272-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6c896735] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6c896735, Job ID: ace7b4c6-cd6e-454c-bf20-160af563101b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:20:03,277-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6c896735] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 92a35c7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:20:03,279-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6c896735] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 92a35c7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:20:03,289-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [783e2f06] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:20:03,290-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [783e2f06] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6770ff86] Autorecovering 2 hosts
>2018-05-29 13:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6770ff86] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:25:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65223807] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:25:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65223807] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:25:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65223807] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:25:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65223807] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:25:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [65223807] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 321d680d
>2018-05-29 13:25:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [65223807] FINISH, SetVdsStatusVDSCommand, log id: 321d680d
>2018-05-29 13:25:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65223807] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:25:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65223807] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:25:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [65223807] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:25:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [68417c15] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:25:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [68417c15] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:25:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [68417c15] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:25:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [68417c15] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:25:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [68417c15] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31af719d
>2018-05-29 13:25:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [68417c15] FINISH, SetVdsStatusVDSCommand, log id: 31af719d
>2018-05-29 13:25:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [68417c15] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 43238b93
>2018-05-29 13:25:00,103-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [68417c15] FINISH, SetHaMaintenanceModeVDSCommand, log id: 43238b93
>2018-05-29 13:25:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [68417c15] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:25:00,105-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [68417c15] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:25:00,844-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [76e2082c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4774093d
>2018-05-29 13:25:00,846-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [76e2082c] FINISH, GetHardwareInfoVDSCommand, log id: 4774093d
>2018-05-29 13:25:00,859-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [1b7611bb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:25:00,868-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [4fa7a4b2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:25:00,873-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [3d56cae5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:25:00,885-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [3d56cae5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 73518694
>2018-05-29 13:25:00,961-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3a278a54] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 666b32a0
>2018-05-29 13:25:00,962-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3a278a54] FINISH, GetHardwareInfoVDSCommand, log id: 666b32a0
>2018-05-29 13:25:00,975-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [342d939b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:25:00,980-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [7af93e49] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:25:00,985-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2f36cac1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:25:00,991-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2f36cac1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 57c05565
>2018-05-29 13:25:01,268-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3d56cae5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:25:01,268-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [3d56cae5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 73518694
>2018-05-29 13:25:01,272-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3d56cae5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:25:01,273-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3d56cae5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:25:01,277-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3d56cae5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:25:01,277-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3d56cae5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:25:01,281-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3d56cae5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:25:01,282-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3d56cae5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:25:01,285-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [3d56cae5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:25:01,288-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3d56cae5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3d56cae5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:25:01,296-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 514096e7
>2018-05-29 13:25:01,296-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:25:01,366-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2f36cac1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:25:01,366-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2f36cac1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 57c05565
>2018-05-29 13:25:01,370-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2f36cac1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:25:01,371-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2f36cac1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:25:01,375-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2f36cac1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:25:01,376-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2f36cac1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:25:01,379-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2f36cac1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:25:01,380-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2f36cac1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:25:01,383-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2f36cac1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:25:01,385-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2f36cac1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2f36cac1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:25:01,393-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:25:01,393-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:25:01,393-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:25:01,393-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:25:01,393-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 514096e7
>2018-05-29 13:25:01,393-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:25:01,399-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 55a72a9c
>2018-05-29 13:25:01,399-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:25:01,409-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [5c5adf35] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:25:01,410-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5c5adf35] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47632cba
>2018-05-29 13:25:01,412-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5c5adf35] FINISH, SetVdsStatusVDSCommand, log id: 47632cba
>2018-05-29 13:25:01,444-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5c5adf35] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5c5adf35, Job ID: 02d4aaf5-3d18-4aef-87bd-4a951b9576c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:25:01,449-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5c5adf35] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4fa7a4b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:25:01,452-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5c5adf35] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4fa7a4b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:25:01,463-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [35e19c82] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:25:01,464-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [35e19c82] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:25:01,499-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:25:01,499-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:25:01,499-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:25:01,499-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:25:01,499-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 55a72a9c
>2018-05-29 13:25:01,499-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:25:01,512-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [59cd03dc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:25:01,514-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [59cd03dc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4a6a6d94
>2018-05-29 13:25:01,516-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [59cd03dc] FINISH, SetVdsStatusVDSCommand, log id: 4a6a6d94
>2018-05-29 13:25:01,521-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59cd03dc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 59cd03dc, Job ID: 398f73b0-f703-4fdc-843d-6125c2386370, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:25:01,526-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59cd03dc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7af93e49, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:25:01,528-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59cd03dc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7af93e49, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:25:01,538-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [3de85557] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:25:01,539-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [3de85557] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [520d9247] Autorecovering 2 hosts
>2018-05-29 13:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [520d9247] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:30:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c222b9b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:30:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c222b9b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:30:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c222b9b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:30:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c222b9b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:30:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5c222b9b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 67a24031
>2018-05-29 13:30:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5c222b9b] FINISH, SetVdsStatusVDSCommand, log id: 67a24031
>2018-05-29 13:30:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c222b9b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:30:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c222b9b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:30:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5c222b9b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:30:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [15d1961f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:30:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [15d1961f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:30:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [15d1961f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:30:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [15d1961f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:30:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [15d1961f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 251764c3
>2018-05-29 13:30:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [15d1961f] FINISH, SetVdsStatusVDSCommand, log id: 251764c3
>2018-05-29 13:30:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [15d1961f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 576d2722
>2018-05-29 13:30:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [15d1961f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 576d2722
>2018-05-29 13:30:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [15d1961f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:30:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [15d1961f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:30:02,125-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [742c88ed] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 16a8c703
>2018-05-29 13:30:02,127-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [742c88ed] FINISH, GetHardwareInfoVDSCommand, log id: 16a8c703
>2018-05-29 13:30:02,140-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [552a4a99] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:30:02,148-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [6a29f41d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:30:02,153-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6772a1b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:30:02,159-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6772a1b8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5f8447e6
>2018-05-29 13:30:02,239-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [5d73035d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3c8ffde5
>2018-05-29 13:30:02,241-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [5d73035d] FINISH, GetHardwareInfoVDSCommand, log id: 3c8ffde5
>2018-05-29 13:30:02,253-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [6346b1b7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:30:02,257-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [594499] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:30:02,262-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [788cd87a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:30:02,268-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [788cd87a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 507517b8
>2018-05-29 13:30:02,536-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6772a1b8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:30:02,536-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6772a1b8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5f8447e6
>2018-05-29 13:30:02,540-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6772a1b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:30:02,540-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6772a1b8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:30:02,544-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6772a1b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:30:02,545-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6772a1b8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:30:02,549-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6772a1b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:30:02,549-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6772a1b8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:30:02,552-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6772a1b8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:30:02,555-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6772a1b8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6772a1b8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:30:02,558-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4aaf4fbc
>2018-05-29 13:30:02,558-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:30:02,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [788cd87a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:30:02,633-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [788cd87a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 507517b8
>2018-05-29 13:30:02,637-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [788cd87a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:30:02,638-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [788cd87a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:30:02,642-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [788cd87a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:30:02,643-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [788cd87a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:30:02,646-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [788cd87a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:30:02,647-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [788cd87a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:30:02,650-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [788cd87a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:30:02,652-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [788cd87a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 788cd87a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:30:02,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:30:02,659-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:30:02,659-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:30:02,659-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:30:02,659-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4aaf4fbc
>2018-05-29 13:30:02,659-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:30:02,661-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6ead0bd2
>2018-05-29 13:30:02,661-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:30:02,672-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [2954da65] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:30:02,674-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2954da65] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46ae5fc1
>2018-05-29 13:30:02,676-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2954da65] FINISH, SetVdsStatusVDSCommand, log id: 46ae5fc1
>2018-05-29 13:30:02,707-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2954da65] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2954da65, Job ID: 7f67a827-3902-474a-a49b-ea92b9e5eda3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:30:02,712-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2954da65] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6a29f41d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:30:02,714-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2954da65] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6a29f41d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:30:02,724-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [5ee18fc1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:30:02,725-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [5ee18fc1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:30:02,759-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:30:02,759-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:30:02,759-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:30:02,759-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:30:02,759-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6ead0bd2
>2018-05-29 13:30:02,759-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:30:02,772-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [71c37238] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:30:02,774-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [71c37238] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 16ada1c6
>2018-05-29 13:30:02,776-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [71c37238] FINISH, SetVdsStatusVDSCommand, log id: 16ada1c6
>2018-05-29 13:30:02,781-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c37238] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 71c37238, Job ID: f7c8cd9a-e810-497e-bcb8-2719c294f224, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:30:02,785-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c37238] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 594499, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:30:02,787-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c37238] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 594499, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:30:02,798-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [2ad8af3d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:30:02,799-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [2ad8af3d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [37b8e118] Autorecovering 2 hosts
>2018-05-29 13:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [37b8e118] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:35:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [8199047] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:35:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [8199047] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:35:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [8199047] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:35:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [8199047] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:35:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [8199047] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 145815ad
>2018-05-29 13:35:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [8199047] FINISH, SetVdsStatusVDSCommand, log id: 145815ad
>2018-05-29 13:35:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [8199047] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:35:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [8199047] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:35:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [8199047] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:35:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17fe7d70] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:35:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17fe7d70] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:35:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17fe7d70] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:35:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17fe7d70] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:35:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [17fe7d70] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 91f3ebb
>2018-05-29 13:35:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [17fe7d70] FINISH, SetVdsStatusVDSCommand, log id: 91f3ebb
>2018-05-29 13:35:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [17fe7d70] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 73bf2ade
>2018-05-29 13:35:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [17fe7d70] FINISH, SetHaMaintenanceModeVDSCommand, log id: 73bf2ade
>2018-05-29 13:35:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17fe7d70] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:35:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17fe7d70] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:35:00,313-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [727490d2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3e1b9c7c
>2018-05-29 13:35:00,315-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [727490d2] FINISH, GetHardwareInfoVDSCommand, log id: 3e1b9c7c
>2018-05-29 13:35:00,327-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [46156d5e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:35:00,331-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [38eb0ec5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:35:00,336-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [5b4c840] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:35:00,390-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [5b4c840] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5edc2f60
>2018-05-29 13:35:00,439-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [34c7385d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5cc7b76e
>2018-05-29 13:35:00,440-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [34c7385d] FINISH, GetHardwareInfoVDSCommand, log id: 5cc7b76e
>2018-05-29 13:35:00,453-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [16d82a01] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:35:00,458-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [2df56b84] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:35:00,463-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [4e9c8989] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:35:00,469-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [4e9c8989] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 63f8347d
>2018-05-29 13:35:00,779-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5b4c840] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:35:00,779-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [5b4c840] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5edc2f60
>2018-05-29 13:35:00,783-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5b4c840] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:35:00,784-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5b4c840] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:35:00,788-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5b4c840] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:35:00,788-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5b4c840] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:35:00,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5b4c840] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:35:00,793-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5b4c840] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:35:00,796-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [5b4c840] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:35:00,798-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5b4c840] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5b4c840, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:35:00,805-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2e319e51
>2018-05-29 13:35:00,805-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:35:00,839-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4e9c8989] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:35:00,839-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [4e9c8989] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 63f8347d
>2018-05-29 13:35:00,842-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4e9c8989] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:35:00,843-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4e9c8989] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:35:00,847-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4e9c8989] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:35:00,847-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4e9c8989] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:35:00,851-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4e9c8989] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:35:00,852-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4e9c8989] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:35:00,854-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [4e9c8989] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:35:00,857-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4e9c8989] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4e9c8989, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:35:00,904-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:35:00,904-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:35:00,904-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:35:00,904-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:35:00,904-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2e319e51
>2018-05-29 13:35:00,904-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:35:00,906-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 44189cb6
>2018-05-29 13:35:00,906-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:35:00,917-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [59058a63] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:35:00,918-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [59058a63] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54613b78
>2018-05-29 13:35:00,920-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [59058a63] FINISH, SetVdsStatusVDSCommand, log id: 54613b78
>2018-05-29 13:35:00,928-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [59058a63] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 59058a63, Job ID: 51c06b48-47ac-48d0-ada3-9b497cdafbe3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:35:00,932-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [59058a63] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 38eb0ec5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:35:00,934-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [59058a63] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 38eb0ec5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:35:00,945-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [7ec3e1c2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:35:00,946-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7ec3e1c2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:35:01,007-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:35:01,007-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:35:01,007-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:35:01,007-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:35:01,007-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 44189cb6
>2018-05-29 13:35:01,007-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:35:01,019-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [5997699] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:35:01,020-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5997699] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 344ce41a
>2018-05-29 13:35:01,022-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5997699] FINISH, SetVdsStatusVDSCommand, log id: 344ce41a
>2018-05-29 13:35:01,028-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5997699] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5997699, Job ID: 332e1f25-9ea9-4887-b61f-a040f1a328c4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:35:01,032-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5997699] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2df56b84, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:35:01,034-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5997699] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2df56b84, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:35:01,044-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [4b7574de] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:35:01,045-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4b7574de] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:40:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [342aa372] Autorecovering 2 hosts
>2018-05-29 13:40:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [342aa372] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:40:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [232993ed] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:40:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [232993ed] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:40:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [232993ed] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:40:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [232993ed] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:40:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [232993ed] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 717317da
>2018-05-29 13:40:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [232993ed] FINISH, SetVdsStatusVDSCommand, log id: 717317da
>2018-05-29 13:40:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [232993ed] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:40:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [232993ed] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:40:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [232993ed] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:40:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [63e26b17] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:40:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [63e26b17] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:40:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [63e26b17] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:40:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [63e26b17] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:40:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [63e26b17] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 75664c8
>2018-05-29 13:40:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [63e26b17] FINISH, SetVdsStatusVDSCommand, log id: 75664c8
>2018-05-29 13:40:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [63e26b17] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 727e2fcb
>2018-05-29 13:40:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [63e26b17] FINISH, SetHaMaintenanceModeVDSCommand, log id: 727e2fcb
>2018-05-29 13:40:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [63e26b17] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:40:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [63e26b17] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:40:01,840-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [292ffb79] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4d340870
>2018-05-29 13:40:01,842-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [292ffb79] FINISH, GetHardwareInfoVDSCommand, log id: 4d340870
>2018-05-29 13:40:01,855-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [14ba6a0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:40:01,864-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [18283ab1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:40:01,869-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2312b9e6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:40:01,876-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2312b9e6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 669fcd54
>2018-05-29 13:40:01,951-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5bec1198] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: f3d7dde
>2018-05-29 13:40:01,953-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5bec1198] FINISH, GetHardwareInfoVDSCommand, log id: f3d7dde
>2018-05-29 13:40:01,968-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [404711c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:40:01,973-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [449dc92d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:40:01,978-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [d007af5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:40:01,985-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [d007af5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 672909c8
>2018-05-29 13:40:02,252-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2312b9e6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:40:02,252-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2312b9e6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 669fcd54
>2018-05-29 13:40:02,256-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2312b9e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:40:02,256-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2312b9e6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:40:02,260-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2312b9e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:40:02,261-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2312b9e6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:40:02,265-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2312b9e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:40:02,266-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2312b9e6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:40:02,270-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2312b9e6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:40:02,272-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2312b9e6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2312b9e6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:40:02,276-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4f5fd662
>2018-05-29 13:40:02,276-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:40:02,353-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d007af5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:40:02,353-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [d007af5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 672909c8
>2018-05-29 13:40:02,356-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d007af5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:40:02,357-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [d007af5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:40:02,361-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d007af5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:40:02,361-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [d007af5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:40:02,365-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d007af5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:40:02,366-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [d007af5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:40:02,370-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [d007af5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:40:02,372-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d007af5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: d007af5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:40:02,375-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:40:02,375-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:40:02,375-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:40:02,375-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:40:02,375-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4f5fd662
>2018-05-29 13:40:02,375-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:40:02,382-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3e0b2bea
>2018-05-29 13:40:02,382-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:40:02,389-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [2142209] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:40:02,391-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2142209] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 473ce6f3
>2018-05-29 13:40:02,392-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2142209] FINISH, SetVdsStatusVDSCommand, log id: 473ce6f3
>2018-05-29 13:40:02,424-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2142209] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2142209, Job ID: 6b607832-b52f-4196-9ced-81deb4b25b43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:40:02,428-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2142209] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 18283ab1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:40:02,430-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2142209] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 18283ab1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:40:02,442-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [6271055d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:40:02,443-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [6271055d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:40:02,481-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:40:02,481-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:40:02,481-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:40:02,481-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:40:02,481-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3e0b2bea
>2018-05-29 13:40:02,481-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:40:02,497-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [3113295a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:40:02,498-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3113295a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 35a72b55
>2018-05-29 13:40:02,501-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3113295a] FINISH, SetVdsStatusVDSCommand, log id: 35a72b55
>2018-05-29 13:40:02,507-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3113295a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3113295a, Job ID: cfcdb6cd-8f34-4c4b-9b1e-4e96939ec9a4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:40:02,511-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3113295a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 449dc92d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:40:02,513-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3113295a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 449dc92d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:40:02,524-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [3f6d6073] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:40:02,525-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [3f6d6073] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1ecf6b32] Autorecovering 2 hosts
>2018-05-29 13:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1ecf6b32] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:45:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7c8b0aa9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7c8b0aa9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7c8b0aa9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7c8b0aa9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:45:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7c8b0aa9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7d7fbbad
>2018-05-29 13:45:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7c8b0aa9] FINISH, SetVdsStatusVDSCommand, log id: 7d7fbbad
>2018-05-29 13:45:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7c8b0aa9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:45:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7c8b0aa9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:45:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7c8b0aa9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:45:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4780ba9d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4780ba9d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4780ba9d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4780ba9d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:45:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4780ba9d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8f1c78
>2018-05-29 13:45:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4780ba9d] FINISH, SetVdsStatusVDSCommand, log id: 8f1c78
>2018-05-29 13:45:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [4780ba9d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1c33ca9b
>2018-05-29 13:45:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [4780ba9d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1c33ca9b
>2018-05-29 13:45:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4780ba9d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:45:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4780ba9d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:45:03,085-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5382a076] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2b3e6273
>2018-05-29 13:45:03,086-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5382a076] FINISH, GetHardwareInfoVDSCommand, log id: 2b3e6273
>2018-05-29 13:45:03,100-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [7e138ea6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:45:03,108-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [715a2615] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:45:03,113-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [13ba5a71] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:45:03,124-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [13ba5a71] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 327768da
>2018-05-29 13:45:03,256-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7fd25ff1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 375c2b23
>2018-05-29 13:45:03,258-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7fd25ff1] FINISH, GetHardwareInfoVDSCommand, log id: 375c2b23
>2018-05-29 13:45:03,270-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [7e9ba8ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:45:03,275-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [3454e54e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:45:03,280-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [210bd937] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:45:03,286-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [210bd937] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 354b231a
>2018-05-29 13:45:03,502-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13ba5a71] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:45:03,502-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [13ba5a71] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 327768da
>2018-05-29 13:45:03,506-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13ba5a71] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:45:03,507-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [13ba5a71] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:45:03,510-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13ba5a71] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:45:03,511-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [13ba5a71] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:45:03,515-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13ba5a71] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:45:03,516-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [13ba5a71] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:45:03,519-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [13ba5a71] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:45:03,521-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13ba5a71] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 13ba5a71, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:45:03,528-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 45c9bbe
>2018-05-29 13:45:03,528-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:45:03,626-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:45:03,626-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:45:03,626-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:45:03,626-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:45:03,626-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 45c9bbe
>2018-05-29 13:45:03,626-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:45:03,639-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [1fd5d347] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:45:03,641-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1fd5d347] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72c637bb
>2018-05-29 13:45:03,643-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1fd5d347] FINISH, SetVdsStatusVDSCommand, log id: 72c637bb
>2018-05-29 13:45:03,660-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [210bd937] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:45:03,660-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [210bd937] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 354b231a
>2018-05-29 13:45:03,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [210bd937] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:45:03,664-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [210bd937] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:45:03,668-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [210bd937] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:45:03,669-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [210bd937] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:45:03,672-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [210bd937] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:45:03,673-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [210bd937] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:45:03,677-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [210bd937] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:45:03,677-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fd5d347] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1fd5d347, Job ID: 248c3d75-0647-4ce4-977b-94befb89d9c2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:45:03,679-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [210bd937] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 210bd937, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:45:03,683-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fd5d347] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 715a2615, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:45:03,683-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 23e12910
>2018-05-29 13:45:03,683-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:45:03,685-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fd5d347] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 715a2615, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:45:03,696-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [4f08a629] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:45:03,698-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [4f08a629] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:45:03,783-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:45:03,783-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:45:03,783-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:45:03,783-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:45:03,783-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 23e12910
>2018-05-29 13:45:03,783-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:45:03,796-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [4841bd82] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:45:03,797-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4841bd82] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4bffaa02
>2018-05-29 13:45:03,799-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4841bd82] FINISH, SetVdsStatusVDSCommand, log id: 4bffaa02
>2018-05-29 13:45:03,831-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4841bd82] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4841bd82, Job ID: ab04b836-3c6a-4bbb-97a3-a5a299c8ead3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:45:03,835-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4841bd82] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3454e54e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:45:03,837-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4841bd82] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3454e54e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:45:03,847-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [63283a12] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:45:03,848-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [63283a12] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2eacf3cc] Autorecovering 2 hosts
>2018-05-29 13:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2eacf3cc] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:50:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [37384b38] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:50:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [37384b38] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:50:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [37384b38] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:50:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [37384b38] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:50:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [37384b38] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54559d3a
>2018-05-29 13:50:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [37384b38] FINISH, SetVdsStatusVDSCommand, log id: 54559d3a
>2018-05-29 13:50:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [37384b38] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:50:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [37384b38] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:50:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [37384b38] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:50:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [14cc66bf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:50:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [14cc66bf] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:50:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [14cc66bf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:50:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [14cc66bf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:50:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [14cc66bf] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 413f7782
>2018-05-29 13:50:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [14cc66bf] FINISH, SetVdsStatusVDSCommand, log id: 413f7782
>2018-05-29 13:50:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [14cc66bf] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 52b291e7
>2018-05-29 13:50:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [14cc66bf] FINISH, SetHaMaintenanceModeVDSCommand, log id: 52b291e7
>2018-05-29 13:50:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [14cc66bf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:50:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [14cc66bf] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:50:01,307-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [37688b44] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 581a1477
>2018-05-29 13:50:01,309-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [37688b44] FINISH, GetHardwareInfoVDSCommand, log id: 581a1477
>2018-05-29 13:50:01,321-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [4370eb9d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:50:01,329-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [7acc00de] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:50:01,334-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [7e584c54] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:50:01,346-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [7e584c54] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: ea38e30
>2018-05-29 13:50:01,430-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3e249911] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 73efd034
>2018-05-29 13:50:01,432-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3e249911] FINISH, GetHardwareInfoVDSCommand, log id: 73efd034
>2018-05-29 13:50:01,445-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [430d3830] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:50:01,454-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3fd5027] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:50:01,459-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [194cdb7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:50:01,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [194cdb7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6dce7166
>2018-05-29 13:50:01,727-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7e584c54] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:50:01,727-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [7e584c54] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: ea38e30
>2018-05-29 13:50:01,731-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7e584c54] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:50:01,732-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7e584c54] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:50:01,736-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7e584c54] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:50:01,736-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7e584c54] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:50:01,740-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7e584c54] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:50:01,741-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7e584c54] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:50:01,744-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [7e584c54] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:50:01,746-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7e584c54] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7e584c54, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:50:01,752-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 61e42f30
>2018-05-29 13:50:01,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:50:01,851-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:50:01,851-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:50:01,851-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:50:01,851-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:50:01,851-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 61e42f30
>2018-05-29 13:50:01,851-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:50:01,864-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [109d56b0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:50:01,866-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [109d56b0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 78bd7680
>2018-05-29 13:50:01,868-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [109d56b0] FINISH, SetVdsStatusVDSCommand, log id: 78bd7680
>2018-05-29 13:50:01,900-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [109d56b0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 109d56b0, Job ID: 459ca66f-6c95-4b92-a77c-f785af63c294, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:50:01,905-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [109d56b0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7acc00de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:50:01,907-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [109d56b0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7acc00de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:50:01,918-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [b7e1086] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:50:01,919-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [b7e1086] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:50:02,879-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [194cdb7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:50:02,879-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [194cdb7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6dce7166
>2018-05-29 13:50:02,882-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [194cdb7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:50:02,883-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [194cdb7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:50:02,887-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [194cdb7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:50:02,888-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [194cdb7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:50:02,891-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [194cdb7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:50:02,892-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [194cdb7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:50:02,895-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [194cdb7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:50:02,898-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [194cdb7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 194cdb7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:50:02,900-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 23d79c8
>2018-05-29 13:50:02,900-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:50:02,997-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:50:02,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:50:02,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:50:02,997-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:50:02,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 23d79c8
>2018-05-29 13:50:02,997-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:50:03,010-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [40d1f2cb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:50:03,011-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [40d1f2cb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26910a6
>2018-05-29 13:50:03,014-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [40d1f2cb] FINISH, SetVdsStatusVDSCommand, log id: 26910a6
>2018-05-29 13:50:03,019-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [40d1f2cb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 40d1f2cb, Job ID: 73388a54-ab2e-4870-8522-c2cadd025afc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:50:03,024-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [40d1f2cb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3fd5027, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:50:03,027-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [40d1f2cb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3fd5027, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:50:03,037-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [27edf4b3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:50:03,038-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [27edf4b3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:54:38,349-04 INFO  [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler2) [12308afc] Backup check started.
>2018-05-29 13:54:38,362-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12308afc] EVENT_ID: ENGINE_NO_FULL_BACKUP(9,022), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: There is no full backup available, please run engine-backup to prevent data loss in case of corruption.
>2018-05-29 13:54:38,362-04 INFO  [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler2) [12308afc] Backup check completed.
>2018-05-29 13:54:43,030-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler2) [29b2ecac] Attempting to update VMs/Templates Ovf.
>2018-05-29 13:54:43,032-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 13:54:43,032-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 13:54:43,033-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:54:43,037-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 13:54:43,041-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 13:54:43,041-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 13:54:43,041-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 13:54:43,041-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 13:54:43,041-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 13:54:43,042-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 13:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7baed0bf] Autorecovering 2 hosts
>2018-05-29 13:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7baed0bf] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:55:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5838a574] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5838a574] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5838a574] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:55:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5838a574] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:55:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5838a574] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 74faa4a7
>2018-05-29 13:55:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5838a574] FINISH, SetVdsStatusVDSCommand, log id: 74faa4a7
>2018-05-29 13:55:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5838a574] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:55:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5838a574] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 13:55:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5838a574] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:55:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62482b1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:55:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62482b1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 13:55:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62482b1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:55:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62482b1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:55:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [62482b1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d5b0752
>2018-05-29 13:55:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [62482b1] FINISH, SetVdsStatusVDSCommand, log id: 3d5b0752
>2018-05-29 13:55:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [62482b1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 17e0a4ba
>2018-05-29 13:55:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [62482b1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 17e0a4ba
>2018-05-29 13:55:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62482b1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 13:55:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62482b1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 13:55:00,669-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [5d682b65] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7f7f3390
>2018-05-29 13:55:00,671-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [5d682b65] FINISH, GetHardwareInfoVDSCommand, log id: 7f7f3390
>2018-05-29 13:55:00,685-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [3d8b8012] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:55:00,693-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [3077ace5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:55:00,698-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3b77bff7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:55:00,704-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3b77bff7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7e064557
>2018-05-29 13:55:01,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3b77bff7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:55:01,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3b77bff7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7e064557
>2018-05-29 13:55:01,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3b77bff7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:55:01,082-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3b77bff7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:55:01,086-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3b77bff7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:55:01,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3b77bff7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:55:01,090-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3b77bff7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:55:01,091-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3b77bff7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:55:01,093-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3b77bff7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:55:01,096-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3b77bff7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3b77bff7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:55:01,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1476fb0d
>2018-05-29 13:55:01,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:55:01,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:55:01,198-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:55:01,198-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:55:01,198-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:55:01,198-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1476fb0d
>2018-05-29 13:55:01,198-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:55:01,212-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2c8d76a3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:55:01,213-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2c8d76a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 227b874
>2018-05-29 13:55:01,215-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2c8d76a3] FINISH, SetVdsStatusVDSCommand, log id: 227b874
>2018-05-29 13:55:01,249-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2c8d76a3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2c8d76a3, Job ID: 583a1db1-7561-46a9-85d1-b8bb57c2372c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:55:01,254-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2c8d76a3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3077ace5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:55:01,255-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2c8d76a3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3077ace5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:55:01,266-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [12743693] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 13:55:01,268-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [12743693] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 13:55:02,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [33a34815] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3dc1a1dd
>2018-05-29 13:55:02,616-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [33a34815] FINISH, GetHardwareInfoVDSCommand, log id: 3dc1a1dd
>2018-05-29 13:55:02,629-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [3502eee7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:55:02,637-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [7b680b65] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:55:02,642-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [3a0e1842] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 13:55:02,648-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [3a0e1842] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 342f83ab
>2018-05-29 13:55:03,034-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3a0e1842] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 13:55:03,034-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [3a0e1842] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 342f83ab
>2018-05-29 13:55:03,038-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3a0e1842] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:55:03,038-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3a0e1842] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:55:03,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3a0e1842] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:55:03,043-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3a0e1842] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:55:03,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3a0e1842] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 13:55:03,048-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3a0e1842] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 13:55:03,050-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [3a0e1842] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 13:55:03,053-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3a0e1842] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3a0e1842, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 13:55:03,059-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2d24545d
>2018-05-29 13:55:03,059-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 13:55:03,161-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:55:03,161-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 13:55:03,161-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 13:55:03,161-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 13:55:03,161-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2d24545d
>2018-05-29 13:55:03,161-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 13:55:03,174-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [6833fcad] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:55:03,175-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6833fcad] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 671ff883
>2018-05-29 13:55:03,177-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6833fcad] FINISH, SetVdsStatusVDSCommand, log id: 671ff883
>2018-05-29 13:55:03,208-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6833fcad] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6833fcad, Job ID: 197a579b-ce3b-4c2c-b709-abdba0934c46, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 13:55:03,213-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6833fcad] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7b680b65, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 13:55:03,215-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6833fcad] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7b680b65, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 13:55:03,226-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [58bbbd12] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 13:55:03,227-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [58bbbd12] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1d0a674] Autorecovering 2 hosts
>2018-05-29 14:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1d0a674] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:00:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2debd8f6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2debd8f6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:00:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2debd8f6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:00:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2debd8f6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:00:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2debd8f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18d1a1f9
>2018-05-29 14:00:00,080-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2debd8f6] FINISH, SetVdsStatusVDSCommand, log id: 18d1a1f9
>2018-05-29 14:00:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2debd8f6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:00:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2debd8f6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:00:00,083-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [2debd8f6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:00:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [498aa64d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [498aa64d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [498aa64d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [498aa64d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:00:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [498aa64d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 384b5bc4
>2018-05-29 14:00:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [498aa64d] FINISH, SetVdsStatusVDSCommand, log id: 384b5bc4
>2018-05-29 14:00:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [498aa64d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5a70e268
>2018-05-29 14:00:00,103-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [498aa64d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5a70e268
>2018-05-29 14:00:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [498aa64d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:00:00,105-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [498aa64d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:00:00,862-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [78c1e55c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 413ba3ff
>2018-05-29 14:00:00,863-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [78c1e55c] FINISH, GetHardwareInfoVDSCommand, log id: 413ba3ff
>2018-05-29 14:00:00,876-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [493e45db] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:00:00,885-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [191c659e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:00:00,890-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [1369d130] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:00:00,902-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [1369d130] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6357682f
>2018-05-29 14:00:01,282-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1369d130] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:00:01,282-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [1369d130] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6357682f
>2018-05-29 14:00:01,286-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1369d130] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:00:01,287-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [1369d130] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:00:01,290-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1369d130] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:00:01,291-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [1369d130] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:00:01,295-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1369d130] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:00:01,295-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [1369d130] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:00:01,298-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [1369d130] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:00:01,301-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1369d130] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1369d130, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:00:01,307-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 155cd37e
>2018-05-29 14:00:01,307-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:00:01,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:00:01,403-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:00:01,403-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:00:01,403-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:00:01,403-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: 155cd37e
>2018-05-29 14:00:01,403-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:00:01,415-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [37ce6bf7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:00:01,417-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [37ce6bf7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ad9fede
>2018-05-29 14:00:01,419-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [37ce6bf7] FINISH, SetVdsStatusVDSCommand, log id: 3ad9fede
>2018-05-29 14:00:01,427-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [37ce6bf7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 37ce6bf7, Job ID: a1e60886-4698-44df-bca2-a7b53426faae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:00:01,431-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [37ce6bf7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 191c659e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:00:01,433-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [37ce6bf7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 191c659e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:00:01,444-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [21da4410] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:00:01,445-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [21da4410] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:00:02,146-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1127ed4d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 67019985
>2018-05-29 14:00:02,148-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1127ed4d] FINISH, GetHardwareInfoVDSCommand, log id: 67019985
>2018-05-29 14:00:02,160-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [37d7897a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:00:02,164-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [687f02d2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:00:02,169-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [26fc55bc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:00:02,223-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [26fc55bc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 608dec95
>2018-05-29 14:00:03,596-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26fc55bc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:00:03,596-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [26fc55bc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 608dec95
>2018-05-29 14:00:03,600-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26fc55bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:00:03,600-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [26fc55bc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:00:03,604-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26fc55bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:00:03,605-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [26fc55bc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:00:03,609-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26fc55bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:00:03,609-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [26fc55bc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:00:03,612-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [26fc55bc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:00:03,615-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26fc55bc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 26fc55bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:00:03,621-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: db5a90a
>2018-05-29 14:00:03,621-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:00:03,718-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:00:03,718-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:00:03,718-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:00:03,718-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:00:03,719-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: db5a90a
>2018-05-29 14:00:03,719-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:00:03,732-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [ab9e32] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:00:03,733-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [ab9e32] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6f457028
>2018-05-29 14:00:03,735-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [ab9e32] FINISH, SetVdsStatusVDSCommand, log id: 6f457028
>2018-05-29 14:00:03,766-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [ab9e32] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: ab9e32, Job ID: 7572ebb5-6bc6-42d9-b45f-a37d080cc583, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:00:03,770-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [ab9e32] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 687f02d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:00:03,772-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [ab9e32] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 687f02d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:00:03,783-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [23d492d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:00:03,784-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [23d492d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6b26f55] Autorecovering 2 hosts
>2018-05-29 14:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6b26f55] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:05:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17f1f870] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17f1f870] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17f1f870] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17f1f870] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:05:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [17f1f870] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 334eba43
>2018-05-29 14:05:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [17f1f870] FINISH, SetVdsStatusVDSCommand, log id: 334eba43
>2018-05-29 14:05:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17f1f870] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:05:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17f1f870] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:05:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [17f1f870] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:05:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [39aff3f0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [39aff3f0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [39aff3f0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [39aff3f0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:05:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [39aff3f0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 74850fff
>2018-05-29 14:05:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [39aff3f0] FINISH, SetVdsStatusVDSCommand, log id: 74850fff
>2018-05-29 14:05:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [39aff3f0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1fff7f90
>2018-05-29 14:05:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [39aff3f0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1fff7f90
>2018-05-29 14:05:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [39aff3f0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:05:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [39aff3f0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:05:01,508-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6d52265a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 39f58d4b
>2018-05-29 14:05:01,510-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6d52265a] FINISH, GetHardwareInfoVDSCommand, log id: 39f58d4b
>2018-05-29 14:05:01,523-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [c6e919e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:05:01,532-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [2815d724] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:05:01,537-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [658dec40] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:05:01,548-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [658dec40] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1b6b560e
>2018-05-29 14:05:01,925-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [658dec40] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:05:01,925-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [658dec40] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1b6b560e
>2018-05-29 14:05:01,929-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [658dec40] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:05:01,929-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [658dec40] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:05:01,933-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [658dec40] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:05:01,934-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [658dec40] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:05:01,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [658dec40] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:05:01,938-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [658dec40] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:05:01,942-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [658dec40] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:05:01,944-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [658dec40] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 658dec40, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:05:01,951-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6d0cc176
>2018-05-29 14:05:01,951-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:05:02,045-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [2ab124d4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 44845f84
>2018-05-29 14:05:02,046-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [2ab124d4] FINISH, GetHardwareInfoVDSCommand, log id: 44845f84
>2018-05-29 14:05:02,050-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:05:02,050-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:05:02,050-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:05:02,050-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:05:02,050-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6d0cc176
>2018-05-29 14:05:02,050-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:05:02,060-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [c70f805] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:05:02,064-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [34b13057] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:05:02,065-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [34b13057] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 454b505f
>2018-05-29 14:05:02,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [34b13057] FINISH, SetVdsStatusVDSCommand, log id: 454b505f
>2018-05-29 14:05:02,069-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [2e86b6e2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:05:02,075-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [342a2d6a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:05:02,087-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [342a2d6a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23c5bbd6
>2018-05-29 14:05:02,100-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [34b13057] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 34b13057, Job ID: 88a822ac-5091-46cf-9bac-2711c71b905a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:05:02,105-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [34b13057] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2815d724, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:05:02,107-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [34b13057] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2815d724, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:05:02,118-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [cb858c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:05:02,119-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [cb858c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:05:02,468-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [342a2d6a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:05:02,468-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [342a2d6a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 23c5bbd6
>2018-05-29 14:05:02,471-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [342a2d6a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:05:02,472-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [342a2d6a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:05:02,476-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [342a2d6a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:05:02,476-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [342a2d6a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:05:02,480-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [342a2d6a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:05:02,481-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [342a2d6a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:05:02,484-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [342a2d6a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:05:02,487-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [342a2d6a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 342a2d6a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:05:02,490-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6035c247
>2018-05-29 14:05:02,490-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:05:02,590-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:05:02,590-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:05:02,590-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:05:02,591-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:05:02,591-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6035c247
>2018-05-29 14:05:02,591-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:05:02,603-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [332efd95] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:05:02,605-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [332efd95] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12761e93
>2018-05-29 14:05:02,607-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [332efd95] FINISH, SetVdsStatusVDSCommand, log id: 12761e93
>2018-05-29 14:05:02,612-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [332efd95] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 332efd95, Job ID: 46d150ad-644b-4e41-9ff9-62dfd3f9d88b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:05:02,616-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [332efd95] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2e86b6e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:05:02,619-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [332efd95] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2e86b6e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:05:02,629-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [140dbee1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:05:02,630-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [140dbee1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:09:38,620-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-1) [393dc732] Connected to host dell-per630-06.css.lab.eng.rdu2.redhat.com with SSH key fingerprint: SHA256:a1hh1jl16Nd9hTtuA/TUOOzVFcsyda1SCcq54arWb6Y
>2018-05-29 14:09:38,624-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-3) [1050a6a8] Connected to host dell-per630-07.css.lab.eng.rdu2.redhat.com with SSH key fingerprint: SHA256:Li7+tEWpDe9k+6GYLAsCWnLuHH60ubhmHCZCQdQb26g
>2018-05-29 14:09:38,627-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-5) [53fae673] Connected to host dell-per630-05.css.lab.eng.rdu2.redhat.com with SSH key fingerprint: SHA256:/j16in72Hste2xxE4lLGG0zCeN8aAMgKVf8TFwHlL7I
>2018-05-29 14:09:38,700-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-1) [393dc732] Installation of dell-per630-06.css.lab.eng.rdu2.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar
>2018-05-29 14:09:38,700-04 INFO  [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (pool-7-thread-1) [393dc732] Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' refresh
>2018-05-29 14:09:38,703-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-3) [1050a6a8] Installation of dell-per630-07.css.lab.eng.rdu2.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar
>2018-05-29 14:09:38,703-04 INFO  [org.ovirt.engine.core.uutils.ssh.SSHDialog] (pool-7-thread-3) [1050a6a8] SSH execute 'root@dell-per630-07.css.lab.eng.rdu2.redhat.com' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True'
>2018-05-29 14:09:38,706-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-5) [53fae673] Installation of dell-per630-05.css.lab.eng.rdu2.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar
>2018-05-29 14:09:38,706-04 INFO  [org.ovirt.engine.core.uutils.ssh.SSHDialog] (pool-7-thread-5) [53fae673] SSH execute 'root@dell-per630-05.css.lab.eng.rdu2.redhat.com' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True'
>2018-05-29 14:09:38,714-04 INFO  [org.ovirt.engine.core.uutils.ssh.SSHDialog] (pool-7-thread-1) [393dc732] SSH execute 'root@dell-per630-06.css.lab.eng.rdu2.redhat.com' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True'
>2018-05-29 14:09:39,738-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Initializing
>2018-05-29 14:09:39,745-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Environment setup
>2018-05-29 14:09:39,756-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Initializing
>2018-05-29 14:09:39,756-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Environment packages setup
>2018-05-29 14:09:39,758-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Initializing
>2018-05-29 14:09:39,762-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Environment setup
>2018-05-29 14:09:39,764-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Environment setup
>2018-05-29 14:09:39,773-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Environment packages setup
>2018-05-29 14:09:39,774-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Environment packages setup
>2018-05-29 14:09:40,349-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Programs detection
>2018-05-29 14:09:40,361-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Programs detection
>2018-05-29 14:09:40,367-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Environment customization
>2018-05-29 14:09:40,368-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Programs detection
>2018-05-29 14:09:40,378-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Logs at host located at: '/tmp/ovirt-host-mgmt-20180529140942-5b44yl.log'
>2018-05-29 14:09:40,379-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Environment customization
>2018-05-29 14:09:40,385-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Environment customization
>2018-05-29 14:09:40,389-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Logs at host located at: '/tmp/ovirt-host-mgmt-20180529140942-iimpip.log'
>2018-05-29 14:09:40,396-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Logs at host located at: '/tmp/ovirt-host-mgmt-20180529140942-7mruz0.log'
>2018-05-29 14:09:40,397-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Setup validation
>2018-05-29 14:09:40,405-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Setup validation
>2018-05-29 14:09:40,408-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Transaction setup
>2018-05-29 14:09:40,418-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Transaction setup
>2018-05-29 14:09:40,426-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Setup validation
>2018-05-29 14:09:40,438-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Transaction setup
>2018-05-29 14:09:40,699-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Misc configuration
>2018-05-29 14:09:40,700-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Package installation
>2018-05-29 14:09:40,714-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Misc configuration
>2018-05-29 14:09:40,714-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Package installation
>2018-05-29 14:09:40,725-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Misc configuration
>2018-05-29 14:09:40,726-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Package installation
>2018-05-29 14:09:51,732-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Misc configuration
>2018-05-29 14:09:51,734-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Transaction commit
>2018-05-29 14:09:51,776-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Misc configuration
>2018-05-29 14:09:51,779-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Transaction commit
>2018-05-29 14:09:51,779-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Misc configuration
>2018-05-29 14:09:51,781-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Transaction commit
>2018-05-29 14:09:52,037-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Closing up
>2018-05-29 14:09:52,039-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Pre-termination
>2018-05-29 14:09:52,048-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-mgmt-20180529140952-dell-per630-07.css.lab.eng.rdu2.redhat.com-1050a6a8.log'
>2018-05-29 14:09:52,086-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Closing up
>2018-05-29 14:09:52,087-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Closing up
>2018-05-29 14:09:52,088-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Pre-termination
>2018-05-29 14:09:52,089-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Pre-termination
>2018-05-29 14:09:52,099-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-mgmt-20180529140952-dell-per630-06.css.lab.eng.rdu2.redhat.com-393dc732.log'
>2018-05-29 14:09:52,100-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-mgmt-20180529140952-dell-per630-05.css.lab.eng.rdu2.redhat.com-53fae673.log'
>2018-05-29 14:09:52,104-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Termination
>2018-05-29 14:09:52,149-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-3) [1050a6a8] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Check for available updates on host dell-per630-07.css.lab.eng.rdu2.redhat.com was completed successfully with message 'no updates found.'.
>2018-05-29 14:09:52,159-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Termination
>2018-05-29 14:09:52,159-04 INFO  [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Termination
>2018-05-29 14:09:52,201-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-1) [393dc732] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Check for available updates on host dell-per630-06.css.lab.eng.rdu2.redhat.com was completed successfully with message 'no updates found.'.
>2018-05-29 14:09:52,212-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-5) [53fae673] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Check for available updates on host dell-per630-05.css.lab.eng.rdu2.redhat.com was completed successfully with message 'no updates found.'.
>2018-05-29 14:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [58e7dfc0] Autorecovering 2 hosts
>2018-05-29 14:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [58e7dfc0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:10:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6dc41352] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6dc41352] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6dc41352] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6dc41352] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:10:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6dc41352] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2baac71
>2018-05-29 14:10:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6dc41352] FINISH, SetVdsStatusVDSCommand, log id: 2baac71
>2018-05-29 14:10:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6dc41352] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:10:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6dc41352] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:10:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6dc41352] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:10:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5c0c2f62] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5c0c2f62] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5c0c2f62] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5c0c2f62] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:10:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5c0c2f62] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: cc0fbb2
>2018-05-29 14:10:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5c0c2f62] FINISH, SetVdsStatusVDSCommand, log id: cc0fbb2
>2018-05-29 14:10:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5c0c2f62] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4c2b82dc
>2018-05-29 14:10:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5c0c2f62] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4c2b82dc
>2018-05-29 14:10:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5c0c2f62] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:10:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5c0c2f62] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:10:00,252-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3d7a3c0b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 50e1481a
>2018-05-29 14:10:00,254-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3d7a3c0b] FINISH, GetHardwareInfoVDSCommand, log id: 50e1481a
>2018-05-29 14:10:00,266-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [6dc56e4d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:10:00,275-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [4dfd790f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:10:00,280-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [320ea1b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:10:00,291-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [320ea1b8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 46363e0c
>2018-05-29 14:10:00,673-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [320ea1b8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:10:00,673-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [320ea1b8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 46363e0c
>2018-05-29 14:10:00,678-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [320ea1b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:10:00,678-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [320ea1b8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:10:00,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [320ea1b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:10:00,683-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [320ea1b8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:10:00,687-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [320ea1b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:10:00,687-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [320ea1b8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:10:00,691-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [320ea1b8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:10:00,694-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [320ea1b8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 320ea1b8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:10:00,697-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 42cac827
>2018-05-29 14:10:00,697-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:10:00,798-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:10:00,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:10:00,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:10:00,798-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:10:00,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 42cac827
>2018-05-29 14:10:00,798-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:10:00,812-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [668ee421] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:10:00,813-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [668ee421] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 519a5f34
>2018-05-29 14:10:00,815-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [668ee421] FINISH, SetVdsStatusVDSCommand, log id: 519a5f34
>2018-05-29 14:10:00,844-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [668ee421] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 668ee421, Job ID: a26d7e58-6ccd-49bf-98df-d6d2c2739034, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:10:00,848-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [668ee421] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4dfd790f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:10:00,850-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [668ee421] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4dfd790f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:10:00,861-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [14ac7a58] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:10:00,862-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [14ac7a58] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:10:02,887-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [12d2e1bf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 10b2dca2
>2018-05-29 14:10:02,889-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [12d2e1bf] FINISH, GetHardwareInfoVDSCommand, log id: 10b2dca2
>2018-05-29 14:10:02,902-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [548d2653] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:10:02,906-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3b23e9da] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:10:02,911-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [637186ce] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:10:02,918-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [637186ce] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f76eb0c
>2018-05-29 14:10:03,284-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [637186ce] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:10:03,284-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [637186ce] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f76eb0c
>2018-05-29 14:10:03,288-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [637186ce] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:10:03,289-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [637186ce] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:10:03,293-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [637186ce] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:10:03,294-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [637186ce] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:10:03,297-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [637186ce] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:10:03,298-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [637186ce] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:10:03,300-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [637186ce] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:10:03,303-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [637186ce] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 637186ce, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:10:03,310-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 13e0063f
>2018-05-29 14:10:03,310-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:10:03,408-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:10:03,408-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:10:03,408-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:10:03,408-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:10:03,408-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 13e0063f
>2018-05-29 14:10:03,408-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:10:03,421-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [7f28a8ea] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:10:03,422-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7f28a8ea] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 14854bfe
>2018-05-29 14:10:03,425-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7f28a8ea] FINISH, SetVdsStatusVDSCommand, log id: 14854bfe
>2018-05-29 14:10:03,457-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f28a8ea] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7f28a8ea, Job ID: d1edd705-16e1-4ca9-bdde-afc5f0123890, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:10:03,462-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f28a8ea] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b23e9da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:10:03,464-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f28a8ea] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b23e9da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:10:03,475-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [348e2f3a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:10:03,476-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [348e2f3a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:15:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2a202684] Autorecovering 2 hosts
>2018-05-29 14:15:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2a202684] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:15:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18bf0693] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18bf0693] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18bf0693] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18bf0693] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:15:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [18bf0693] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 403d94a6
>2018-05-29 14:15:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [18bf0693] FINISH, SetVdsStatusVDSCommand, log id: 403d94a6
>2018-05-29 14:15:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18bf0693] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:15:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18bf0693] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:15:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [18bf0693] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:15:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2d22c176] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2d22c176] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2d22c176] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2d22c176] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:15:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2d22c176] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 21709b7c
>2018-05-29 14:15:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2d22c176] FINISH, SetVdsStatusVDSCommand, log id: 21709b7c
>2018-05-29 14:15:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2d22c176] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 678deffc
>2018-05-29 14:15:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2d22c176] FINISH, SetHaMaintenanceModeVDSCommand, log id: 678deffc
>2018-05-29 14:15:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2d22c176] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:15:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2d22c176] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:15:01,164-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [33a473b5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6aafcf5d
>2018-05-29 14:15:01,166-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [33a473b5] FINISH, GetHardwareInfoVDSCommand, log id: 6aafcf5d
>2018-05-29 14:15:01,179-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [66602826] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:15:01,187-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [6a5f6ed] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:15:01,192-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [492cf03] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:15:01,203-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [492cf03] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 51527c33
>2018-05-29 14:15:01,458-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [205803a1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6445facc
>2018-05-29 14:15:01,460-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [205803a1] FINISH, GetHardwareInfoVDSCommand, log id: 6445facc
>2018-05-29 14:15:01,472-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [1a5977ef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:15:01,476-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [3d927f47] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:15:01,481-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [30d6c0ab] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:15:01,487-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [30d6c0ab] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 292bae1e
>2018-05-29 14:15:01,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [492cf03] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:15:01,575-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [492cf03] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 51527c33
>2018-05-29 14:15:01,579-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [492cf03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:15:01,580-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [492cf03] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:15:01,583-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [492cf03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:15:01,584-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [492cf03] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:15:01,588-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [492cf03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:15:01,589-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [492cf03] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:15:01,591-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [492cf03] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:15:01,594-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [492cf03] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 492cf03, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:15:01,600-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4416eed
>2018-05-29 14:15:01,600-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:15:01,698-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:15:01,698-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:15:01,698-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:15:01,698-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:15:01,698-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4416eed
>2018-05-29 14:15:01,698-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:15:01,711-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [243823f6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:15:01,713-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [243823f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2dcbe46b
>2018-05-29 14:15:01,715-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [243823f6] FINISH, SetVdsStatusVDSCommand, log id: 2dcbe46b
>2018-05-29 14:15:01,723-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [243823f6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 243823f6, Job ID: 4d271992-1b41-4fa5-b469-54be081d8e3d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:15:01,728-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [243823f6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6a5f6ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:15:01,730-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [243823f6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6a5f6ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:15:01,740-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [4a8bd8c5] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:15:01,741-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [4a8bd8c5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:15:01,873-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30d6c0ab] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:15:01,873-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [30d6c0ab] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 292bae1e
>2018-05-29 14:15:01,877-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30d6c0ab] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:15:01,877-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [30d6c0ab] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:15:01,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30d6c0ab] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:15:01,882-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [30d6c0ab] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:15:01,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30d6c0ab] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:15:01,886-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [30d6c0ab] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:15:01,889-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [30d6c0ab] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:15:01,892-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30d6c0ab] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 30d6c0ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:15:01,898-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 32521552
>2018-05-29 14:15:01,898-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:15:02,001-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:15:02,001-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:15:02,001-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:15:02,001-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:15:02,001-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 32521552
>2018-05-29 14:15:02,001-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:15:02,013-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [70b25635] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:15:02,015-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [70b25635] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 531be07c
>2018-05-29 14:15:02,017-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [70b25635] FINISH, SetVdsStatusVDSCommand, log id: 531be07c
>2018-05-29 14:15:02,022-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70b25635] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 70b25635, Job ID: 2c8b197c-62d2-48b9-bfb6-3e62cfa8e225, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:15:02,027-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70b25635] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3d927f47, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:15:02,029-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70b25635] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3d927f47, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:15:02,039-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [4f690dd6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:15:02,041-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4f690dd6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3727c455] Autorecovering 2 hosts
>2018-05-29 14:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3727c455] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:20:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [171bac7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:20:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [171bac7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:20:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [171bac7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:20:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [171bac7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:20:00,051-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [171bac7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: bbea172
>2018-05-29 14:20:00,053-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [171bac7] FINISH, SetVdsStatusVDSCommand, log id: bbea172
>2018-05-29 14:20:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [171bac7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:20:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [171bac7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:20:00,056-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [171bac7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:20:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3277baf1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:20:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3277baf1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:20:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3277baf1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:20:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3277baf1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:20:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3277baf1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47c6af9e
>2018-05-29 14:20:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3277baf1] FINISH, SetVdsStatusVDSCommand, log id: 47c6af9e
>2018-05-29 14:20:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [3277baf1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5c9b81d1
>2018-05-29 14:20:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [3277baf1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5c9b81d1
>2018-05-29 14:20:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3277baf1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:20:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3277baf1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:20:02,756-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [20a5b307] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 742fe286
>2018-05-29 14:20:02,758-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [20a5b307] FINISH, GetHardwareInfoVDSCommand, log id: 742fe286
>2018-05-29 14:20:02,771-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [f16c3c2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:20:02,780-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [6bf88d1a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:20:02,785-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [12af08bf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:20:02,796-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [12af08bf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4a64d31b
>2018-05-29 14:20:02,915-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5aecfec7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 320b757b
>2018-05-29 14:20:02,917-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5aecfec7] FINISH, GetHardwareInfoVDSCommand, log id: 320b757b
>2018-05-29 14:20:02,928-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [39ebc155] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:20:02,933-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [5f4fad2e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:20:02,938-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [170be02b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:20:02,943-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [170be02b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 79b3d285
>2018-05-29 14:20:03,326-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [170be02b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:20:03,326-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [170be02b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 79b3d285
>2018-05-29 14:20:03,330-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [170be02b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:20:03,331-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [170be02b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:20:03,335-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [170be02b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:20:03,336-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [170be02b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:20:03,339-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [170be02b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:20:03,340-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [170be02b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:20:03,343-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [170be02b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:20:03,345-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [170be02b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 170be02b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:20:03,348-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 273df5bb
>2018-05-29 14:20:03,348-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:20:03,447-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:20:03,447-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:20:03,447-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:20:03,447-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:20:03,447-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 273df5bb
>2018-05-29 14:20:03,447-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:20:03,460-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [60bcb125] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:20:03,462-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [60bcb125] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5dd7e83c
>2018-05-29 14:20:03,463-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [60bcb125] FINISH, SetVdsStatusVDSCommand, log id: 5dd7e83c
>2018-05-29 14:20:03,492-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [60bcb125] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 60bcb125, Job ID: ce7de919-155d-4ca7-82a0-55247d933912, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:20:03,497-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [60bcb125] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5f4fad2e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:20:03,499-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [60bcb125] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5f4fad2e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:20:03,509-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [66470f69] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:20:03,510-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [66470f69] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:20:04,165-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12af08bf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:20:04,165-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [12af08bf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4a64d31b
>2018-05-29 14:20:04,169-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12af08bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:20:04,170-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [12af08bf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:20:04,174-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12af08bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:20:04,175-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [12af08bf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:20:04,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12af08bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:20:04,179-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [12af08bf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:20:04,182-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [12af08bf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:20:04,185-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12af08bf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 12af08bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:20:04,188-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3700bf46
>2018-05-29 14:20:04,188-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:20:04,287-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:20:04,287-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:20:04,287-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:20:04,287-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:20:04,287-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3700bf46
>2018-05-29 14:20:04,287-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:20:04,299-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [cae50b9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:20:04,300-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [cae50b9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 69861115
>2018-05-29 14:20:04,302-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [cae50b9] FINISH, SetVdsStatusVDSCommand, log id: 69861115
>2018-05-29 14:20:04,308-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [cae50b9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: cae50b9, Job ID: 8363ee7e-e818-4e10-b432-fae9b328ad06, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:20:04,312-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [cae50b9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6bf88d1a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:20:04,314-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [cae50b9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6bf88d1a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:20:04,325-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [481de46f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:20:04,326-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [481de46f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2d275340] Autorecovering 2 hosts
>2018-05-29 14:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2d275340] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:25:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29631fb2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:25:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29631fb2] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:25:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29631fb2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:25:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29631fb2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:25:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [29631fb2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 875d59
>2018-05-29 14:25:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [29631fb2] FINISH, SetVdsStatusVDSCommand, log id: 875d59
>2018-05-29 14:25:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29631fb2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:25:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29631fb2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:25:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [29631fb2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:25:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58938c55] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:25:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58938c55] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:25:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58938c55] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:25:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58938c55] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:25:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [58938c55] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6e8c502
>2018-05-29 14:25:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [58938c55] FINISH, SetVdsStatusVDSCommand, log id: 6e8c502
>2018-05-29 14:25:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [58938c55] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 75fdb3b7
>2018-05-29 14:25:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [58938c55] FINISH, SetHaMaintenanceModeVDSCommand, log id: 75fdb3b7
>2018-05-29 14:25:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58938c55] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:25:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58938c55] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:25:01,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [369680fa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 493c3800
>2018-05-29 14:25:01,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [369680fa] FINISH, GetHardwareInfoVDSCommand, log id: 493c3800
>2018-05-29 14:25:01,085-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6b227e75] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:25:01,093-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [711c75ca] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:25:01,098-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5b5f80f5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:25:01,110-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5b5f80f5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7544b9f5
>2018-05-29 14:25:01,494-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5b5f80f5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:25:01,494-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5b5f80f5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7544b9f5
>2018-05-29 14:25:01,497-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5b5f80f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:25:01,498-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5b5f80f5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:25:01,502-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5b5f80f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:25:01,503-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5b5f80f5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:25:01,506-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5b5f80f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:25:01,507-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5b5f80f5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:25:01,510-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5b5f80f5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:25:01,512-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5b5f80f5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5b5f80f5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:25:01,519-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 66b92ab5
>2018-05-29 14:25:01,519-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:25:01,621-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:25:01,621-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:25:01,621-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:25:01,621-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:25:01,621-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 66b92ab5
>2018-05-29 14:25:01,621-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:25:01,634-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [4a8120fd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:25:01,635-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4a8120fd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3e146537
>2018-05-29 14:25:01,638-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4a8120fd] FINISH, SetVdsStatusVDSCommand, log id: 3e146537
>2018-05-29 14:25:01,669-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4a8120fd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4a8120fd, Job ID: f6c250f9-aefe-46df-87f8-309187741c40, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:25:01,674-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4a8120fd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 711c75ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:25:01,676-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4a8120fd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 711c75ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:25:01,687-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [30574dd4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:25:01,688-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [30574dd4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:25:02,046-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [16be73e6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 62b16eed
>2018-05-29 14:25:02,047-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [16be73e6] FINISH, GetHardwareInfoVDSCommand, log id: 62b16eed
>2018-05-29 14:25:02,060-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [5211e126] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:25:02,068-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [105a3ead] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:25:02,073-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4c37ae51] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:25:02,085-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4c37ae51] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 590c24f6
>2018-05-29 14:25:02,461-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c37ae51] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:25:02,461-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4c37ae51] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 590c24f6
>2018-05-29 14:25:02,465-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c37ae51] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:25:02,466-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4c37ae51] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:25:02,470-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c37ae51] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:25:02,470-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4c37ae51] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:25:02,474-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c37ae51] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:25:02,475-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4c37ae51] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:25:02,478-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4c37ae51] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:25:02,480-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c37ae51] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4c37ae51, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:25:02,487-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 38944f92
>2018-05-29 14:25:02,487-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:25:02,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:25:02,586-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:25:02,586-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:25:02,586-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:25:02,586-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 38944f92
>2018-05-29 14:25:02,586-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:25:02,600-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [3ef11697] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:25:02,601-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3ef11697] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39af0a9b
>2018-05-29 14:25:02,603-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3ef11697] FINISH, SetVdsStatusVDSCommand, log id: 39af0a9b
>2018-05-29 14:25:02,636-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3ef11697] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3ef11697, Job ID: 756ad5ba-1a51-4007-a981-76888612dfe5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:25:02,641-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3ef11697] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 105a3ead, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:25:02,643-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3ef11697] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 105a3ead, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:25:02,654-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [5fdfea96] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:25:02,655-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [5fdfea96] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3c63f86e] Autorecovering 2 hosts
>2018-05-29 14:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3c63f86e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:30:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [505cf410] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:30:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [505cf410] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:30:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [505cf410] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:30:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [505cf410] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:30:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [505cf410] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2aa5cb41
>2018-05-29 14:30:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [505cf410] FINISH, SetVdsStatusVDSCommand, log id: 2aa5cb41
>2018-05-29 14:30:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [505cf410] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:30:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [505cf410] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:30:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [505cf410] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:30:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [45fe30d2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:30:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [45fe30d2] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:30:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [45fe30d2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:30:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [45fe30d2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:30:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [45fe30d2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47649797
>2018-05-29 14:30:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [45fe30d2] FINISH, SetVdsStatusVDSCommand, log id: 47649797
>2018-05-29 14:30:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [45fe30d2] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 612f2126
>2018-05-29 14:30:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [45fe30d2] FINISH, SetHaMaintenanceModeVDSCommand, log id: 612f2126
>2018-05-29 14:30:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [45fe30d2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:30:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [45fe30d2] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:30:00,366-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [13d5905e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5628f673
>2018-05-29 14:30:00,368-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [13d5905e] FINISH, GetHardwareInfoVDSCommand, log id: 5628f673
>2018-05-29 14:30:00,380-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [4b6c88d1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:30:00,389-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [4932a413] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:30:00,394-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [27a5d96b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:30:00,406-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [27a5d96b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 769286ad
>2018-05-29 14:30:00,787-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27a5d96b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:30:00,787-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [27a5d96b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 769286ad
>2018-05-29 14:30:00,791-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27a5d96b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:30:00,792-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [27a5d96b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:30:00,796-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27a5d96b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:30:00,796-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [27a5d96b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:30:00,800-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27a5d96b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:30:00,801-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [27a5d96b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:30:00,804-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [27a5d96b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:30:00,806-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27a5d96b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 27a5d96b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:30:00,813-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 64f17ad6
>2018-05-29 14:30:00,813-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:30:00,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:30:00,912-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:30:00,912-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:30:00,912-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:30:00,912-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 64f17ad6
>2018-05-29 14:30:00,912-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:30:00,925-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4cd8f0d6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:30:00,926-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4cd8f0d6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7ad5136e
>2018-05-29 14:30:00,928-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4cd8f0d6] FINISH, SetVdsStatusVDSCommand, log id: 7ad5136e
>2018-05-29 14:30:00,960-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4cd8f0d6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4cd8f0d6, Job ID: 04bb2714-7397-413a-96b5-d910f85e7cc8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:30:00,964-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4cd8f0d6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4932a413, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:30:00,966-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4cd8f0d6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4932a413, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:30:00,977-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [3b701538] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:30:00,978-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [3b701538] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:30:02,232-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [d94a14b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6bba92a6
>2018-05-29 14:30:02,234-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [d94a14b] FINISH, GetHardwareInfoVDSCommand, log id: 6bba92a6
>2018-05-29 14:30:02,246-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [52836f3e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:30:02,255-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [29903996] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:30:02,260-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [53659830] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:30:02,271-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [53659830] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7264cb6c
>2018-05-29 14:30:02,650-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [53659830] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:30:02,650-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [53659830] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7264cb6c
>2018-05-29 14:30:02,655-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [53659830] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:30:02,655-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [53659830] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:30:02,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [53659830] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:30:02,660-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [53659830] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:30:02,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [53659830] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:30:02,665-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [53659830] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:30:02,667-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [53659830] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:30:02,670-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [53659830] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 53659830, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:30:02,676-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 35014710
>2018-05-29 14:30:02,676-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:30:02,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:30:02,773-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:30:02,773-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:30:02,773-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:30:02,773-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 35014710
>2018-05-29 14:30:02,773-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:30:02,786-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [1ea27a5d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:30:02,787-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1ea27a5d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 20f69683
>2018-05-29 14:30:02,789-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1ea27a5d] FINISH, SetVdsStatusVDSCommand, log id: 20f69683
>2018-05-29 14:30:02,821-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1ea27a5d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1ea27a5d, Job ID: 759a1207-f58b-4f91-b42c-e15df233a4f7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:30:02,825-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1ea27a5d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29903996, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:30:02,827-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1ea27a5d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29903996, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:30:02,838-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [5f7193e6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:30:02,839-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [5f7193e6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6c1958d6] Autorecovering 2 hosts
>2018-05-29 14:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6c1958d6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:35:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [680d47fb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:35:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [680d47fb] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:35:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [680d47fb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:35:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [680d47fb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:35:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [680d47fb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6f3c145e
>2018-05-29 14:35:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [680d47fb] FINISH, SetVdsStatusVDSCommand, log id: 6f3c145e
>2018-05-29 14:35:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [680d47fb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:35:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [680d47fb] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:35:00,082-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [680d47fb] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:35:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [73fa1e20] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:35:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [73fa1e20] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:35:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [73fa1e20] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:35:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [73fa1e20] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:35:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [73fa1e20] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ccee491
>2018-05-29 14:35:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [73fa1e20] FINISH, SetVdsStatusVDSCommand, log id: 3ccee491
>2018-05-29 14:35:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [73fa1e20] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 159ba4da
>2018-05-29 14:35:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [73fa1e20] FINISH, SetHaMaintenanceModeVDSCommand, log id: 159ba4da
>2018-05-29 14:35:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [73fa1e20] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:35:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [73fa1e20] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:35:00,404-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [5f08077] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3e8d707a
>2018-05-29 14:35:00,406-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [5f08077] FINISH, GetHardwareInfoVDSCommand, log id: 3e8d707a
>2018-05-29 14:35:00,419-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [6f6facd8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:35:00,427-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4bdc3416] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:35:00,432-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [222e6b94] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:35:00,439-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [222e6b94] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 85849c4
>2018-05-29 14:35:00,812-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [222e6b94] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:35:00,812-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [222e6b94] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 85849c4
>2018-05-29 14:35:00,816-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [222e6b94] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:35:00,817-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [222e6b94] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:35:00,821-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [222e6b94] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:35:00,821-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [222e6b94] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:35:00,825-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [222e6b94] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:35:00,826-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [222e6b94] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:35:00,829-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [222e6b94] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:35:00,832-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [222e6b94] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 222e6b94, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:35:00,835-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1d2c66c
>2018-05-29 14:35:00,835-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:35:00,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:35:00,935-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:35:00,935-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:35:00,935-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:35:00,935-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1d2c66c
>2018-05-29 14:35:00,935-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:35:00,948-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [44a6ad0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:35:00,949-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [44a6ad0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 78dde6d4
>2018-05-29 14:35:00,951-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [44a6ad0] FINISH, SetVdsStatusVDSCommand, log id: 78dde6d4
>2018-05-29 14:35:00,982-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44a6ad0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 44a6ad0, Job ID: 4c94387b-3bca-46bb-9f93-98926d983a31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:35:00,987-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44a6ad0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4bdc3416, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:35:00,989-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44a6ad0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4bdc3416, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:35:01,000-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [2f2a002b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:35:01,001-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [2f2a002b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:35:01,689-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [34cc619b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 414b897d
>2018-05-29 14:35:01,690-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [34cc619b] FINISH, GetHardwareInfoVDSCommand, log id: 414b897d
>2018-05-29 14:35:01,705-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [265f44f0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:35:01,714-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [3fd7a2f3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:35:01,720-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [41e0ad92] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:35:01,732-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [41e0ad92] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5cfb89d0
>2018-05-29 14:35:02,110-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41e0ad92] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:35:02,110-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [41e0ad92] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5cfb89d0
>2018-05-29 14:35:02,114-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41e0ad92] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:35:02,115-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [41e0ad92] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:35:02,119-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41e0ad92] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:35:02,120-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [41e0ad92] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:35:02,124-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41e0ad92] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:35:02,124-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [41e0ad92] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:35:02,128-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [41e0ad92] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:35:02,130-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41e0ad92] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 41e0ad92, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:35:02,133-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 51f016a0
>2018-05-29 14:35:02,133-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:35:02,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:35:02,230-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:35:02,230-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:35:02,230-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:35:02,230-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 51f016a0
>2018-05-29 14:35:02,230-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:35:02,244-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1445b4c1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:35:02,245-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1445b4c1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3562a9d7
>2018-05-29 14:35:02,247-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1445b4c1] FINISH, SetVdsStatusVDSCommand, log id: 3562a9d7
>2018-05-29 14:35:02,277-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1445b4c1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1445b4c1, Job ID: 0d468c1c-a721-4bac-bf96-f7b3a705a18c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:35:02,281-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1445b4c1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3fd7a2f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:35:02,283-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1445b4c1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3fd7a2f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:35:02,294-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [2fd078a9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:35:02,295-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [2fd078a9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:40:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [30d191c8] Autorecovering 2 hosts
>2018-05-29 14:40:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [30d191c8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:40:00,035-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6196e1d2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:40:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6196e1d2] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:40:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6196e1d2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:40:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6196e1d2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:40:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6196e1d2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 430131ce
>2018-05-29 14:40:00,047-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6196e1d2] FINISH, SetVdsStatusVDSCommand, log id: 430131ce
>2018-05-29 14:40:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6196e1d2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:40:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6196e1d2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:40:00,050-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [6196e1d2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:40:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c53c53f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:40:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c53c53f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:40:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c53c53f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:40:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c53c53f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:40:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5c53c53f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3a6fa248
>2018-05-29 14:40:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5c53c53f] FINISH, SetVdsStatusVDSCommand, log id: 3a6fa248
>2018-05-29 14:40:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5c53c53f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3edaf3d6
>2018-05-29 14:40:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5c53c53f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3edaf3d6
>2018-05-29 14:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c53c53f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:40:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c53c53f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:40:01,828-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3291650a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4d123488
>2018-05-29 14:40:01,829-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3291650a] FINISH, GetHardwareInfoVDSCommand, log id: 4d123488
>2018-05-29 14:40:01,843-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [65f8e85c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:40:01,852-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [73ca19d3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:40:01,858-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [30113b7e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:40:01,869-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [30113b7e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 114c2b66
>2018-05-29 14:40:02,248-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [30113b7e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:40:02,248-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [30113b7e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 114c2b66
>2018-05-29 14:40:02,252-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [30113b7e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:40:02,252-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [30113b7e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:40:02,256-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [30113b7e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:40:02,257-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [30113b7e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:40:02,261-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [30113b7e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:40:02,261-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [30113b7e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:40:02,264-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [30113b7e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:40:02,267-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [30113b7e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 30113b7e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:40:02,273-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1d461d52
>2018-05-29 14:40:02,273-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:40:02,374-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:40:02,374-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:40:02,374-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:40:02,374-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:40:02,374-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1d461d52
>2018-05-29 14:40:02,374-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:40:02,388-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [2956e6e3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:40:02,389-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2956e6e3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6a97b1ca
>2018-05-29 14:40:02,391-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2956e6e3] FINISH, SetVdsStatusVDSCommand, log id: 6a97b1ca
>2018-05-29 14:40:02,399-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2956e6e3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2956e6e3, Job ID: 4bda97b5-476e-4c0b-9b17-f77eb3ebe388, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:40:02,404-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2956e6e3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 73ca19d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:40:02,406-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2956e6e3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 73ca19d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:40:02,418-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [51bec90] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:40:02,419-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [51bec90] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:40:03,067-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2b1b6052] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 27ae1b0f
>2018-05-29 14:40:03,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2b1b6052] FINISH, GetHardwareInfoVDSCommand, log id: 27ae1b0f
>2018-05-29 14:40:03,083-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [ebd033c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:40:03,088-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1c206c31] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:40:03,093-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1d99a9e4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:40:03,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1d99a9e4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2b60f0ec
>2018-05-29 14:40:05,424-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d99a9e4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:40:05,424-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1d99a9e4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2b60f0ec
>2018-05-29 14:40:05,428-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d99a9e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:40:05,429-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1d99a9e4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:40:05,433-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d99a9e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:40:05,434-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1d99a9e4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:40:05,438-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d99a9e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:40:05,439-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1d99a9e4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:40:05,442-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1d99a9e4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:40:05,445-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d99a9e4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1d99a9e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:40:05,448-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1206e0f4
>2018-05-29 14:40:05,448-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:40:05,543-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:40:05,543-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:40:05,543-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:40:05,543-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:40:05,543-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1206e0f4
>2018-05-29 14:40:05,544-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:40:05,557-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [b65b53c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:40:05,558-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [b65b53c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ed887a8
>2018-05-29 14:40:05,560-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [b65b53c] FINISH, SetVdsStatusVDSCommand, log id: 2ed887a8
>2018-05-29 14:40:05,566-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [b65b53c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: b65b53c, Job ID: 5587a1e7-0f85-493f-aa42-e623491df52a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:40:05,571-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [b65b53c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1c206c31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:40:05,574-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [b65b53c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1c206c31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:40:05,584-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [edc81c1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:40:05,585-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [edc81c1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4b75f368] Autorecovering 2 hosts
>2018-05-29 14:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4b75f368] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:45:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1680a402] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:45:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1680a402] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:45:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1680a402] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:45:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1680a402] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:45:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1680a402] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 79d343b2
>2018-05-29 14:45:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1680a402] FINISH, SetVdsStatusVDSCommand, log id: 79d343b2
>2018-05-29 14:45:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1680a402] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:45:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1680a402] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:45:00,102-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1680a402] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:45:00,108-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3b3e0edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:45:00,114-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3b3e0edb] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:45:00,114-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3b3e0edb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:45:00,114-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3b3e0edb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:45:00,116-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3b3e0edb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ed8c0c7
>2018-05-29 14:45:00,118-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3b3e0edb] FINISH, SetVdsStatusVDSCommand, log id: ed8c0c7
>2018-05-29 14:45:00,120-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3b3e0edb] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 53452f5b
>2018-05-29 14:45:00,122-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3b3e0edb] FINISH, SetHaMaintenanceModeVDSCommand, log id: 53452f5b
>2018-05-29 14:45:00,122-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3b3e0edb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:45:00,126-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3b3e0edb] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 3b3e0edb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-29 14:45:00,127-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3b3e0edb] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:45:03,966-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [fb1d2ab] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: ab0fa49
>2018-05-29 14:45:03,967-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [fb1d2ab] FINISH, GetHardwareInfoVDSCommand, log id: ab0fa49
>2018-05-29 14:45:03,976-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7bc0206b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 23cd0ca
>2018-05-29 14:45:03,978-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7bc0206b] FINISH, GetHardwareInfoVDSCommand, log id: 23cd0ca
>2018-05-29 14:45:03,982-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [648d8eba] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:45:03,991-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [ae7ec1f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:45:03,991-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [75fa2d04] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:45:03,996-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [8b77312] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:45:04,000-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2effacc3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:45:04,005-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [8d8f58b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:45:04,009-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [8b77312] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3133e56b
>2018-05-29 14:45:04,012-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [8d8f58b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 393222ee
>2018-05-29 14:45:04,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8d8f58b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:45:04,388-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [8d8f58b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 393222ee
>2018-05-29 14:45:04,393-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8d8f58b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:45:04,394-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [8d8f58b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:45:04,394-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8b77312] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:45:04,394-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [8b77312] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3133e56b
>2018-05-29 14:45:04,398-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8d8f58b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:45:04,398-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [8d8f58b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:45:04,399-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8b77312] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:45:04,400-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [8b77312] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:45:04,402-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8d8f58b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:45:04,403-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [8d8f58b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:45:04,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8b77312] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:45:04,404-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [8b77312] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:45:04,406-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [8d8f58b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:45:04,408-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8b77312] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:45:04,409-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [8b77312] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:45:04,410-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8d8f58b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 8d8f58b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:45:04,413-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [8b77312] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:45:04,413-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2697a9ff
>2018-05-29 14:45:04,413-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:45:04,415-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8b77312] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 8b77312, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:45:04,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:45:04,511-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:45:04,511-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:45:04,511-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:45:04,511-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2697a9ff
>2018-05-29 14:45:04,511-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:45:04,517-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 28334ec8
>2018-05-29 14:45:04,517-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:45:04,525-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [47527ea4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:45:04,526-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [47527ea4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 777a9af5
>2018-05-29 14:45:04,529-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [47527ea4] FINISH, SetVdsStatusVDSCommand, log id: 777a9af5
>2018-05-29 14:45:04,559-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [47527ea4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 47527ea4, Job ID: 14f2f45c-27f0-46e4-bdf9-5110192830ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:45:04,564-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [47527ea4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2effacc3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:45:04,566-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [47527ea4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2effacc3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:45:04,577-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [7922e4c9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:45:04,578-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [7922e4c9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:45:04,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:45:04,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:45:04,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:45:04,614-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:45:04,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 28334ec8
>2018-05-29 14:45:04,614-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:45:04,627-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [24875b2e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:45:04,628-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [24875b2e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4cf1b8e0
>2018-05-29 14:45:04,630-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [24875b2e] FINISH, SetVdsStatusVDSCommand, log id: 4cf1b8e0
>2018-05-29 14:45:04,635-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24875b2e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 24875b2e, Job ID: c56eabf1-2e51-42a5-aec3-efc285d34ffc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:45:04,640-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24875b2e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: ae7ec1f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:45:04,642-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24875b2e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: ae7ec1f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:45:04,652-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [55b2719f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:45:04,653-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [55b2719f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [654d546c] Autorecovering 2 hosts
>2018-05-29 14:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [654d546c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:50:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78c4993e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:50:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78c4993e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:50:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78c4993e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:50:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78c4993e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:50:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78c4993e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4244ff10
>2018-05-29 14:50:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78c4993e] FINISH, SetVdsStatusVDSCommand, log id: 4244ff10
>2018-05-29 14:50:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78c4993e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:50:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78c4993e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:50:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [78c4993e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:50:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [629bb0f8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:50:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [629bb0f8] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:50:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [629bb0f8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:50:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [629bb0f8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:50:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [629bb0f8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b89edab
>2018-05-29 14:50:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [629bb0f8] FINISH, SetVdsStatusVDSCommand, log id: 3b89edab
>2018-05-29 14:50:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [629bb0f8] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3cf5b618
>2018-05-29 14:50:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [629bb0f8] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3cf5b618
>2018-05-29 14:50:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [629bb0f8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:50:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [629bb0f8] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:50:02,237-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4c8a2d47] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 216f9c25
>2018-05-29 14:50:02,238-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4c8a2d47] FINISH, GetHardwareInfoVDSCommand, log id: 216f9c25
>2018-05-29 14:50:02,251-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1d5eaa60] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 64c7b637
>2018-05-29 14:50:02,252-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1d5eaa60] FINISH, GetHardwareInfoVDSCommand, log id: 64c7b637
>2018-05-29 14:50:02,253-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [7fd90600] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:50:02,263-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [60c4d016] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:50:02,265-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [4a14ea84] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:50:02,268-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [23d8290f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:50:02,270-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [43852e18] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:50:02,275-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [23d8290f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3784ffd4
>2018-05-29 14:50:02,276-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [417668fa] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:50:02,282-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [417668fa] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 382b8909
>2018-05-29 14:50:02,658-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [23d8290f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:50:02,659-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [23d8290f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3784ffd4
>2018-05-29 14:50:02,660-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [417668fa] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:50:02,660-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [417668fa] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 382b8909
>2018-05-29 14:50:02,663-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [23d8290f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:50:02,664-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [23d8290f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:50:02,667-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [417668fa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:50:02,668-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [417668fa] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:50:02,669-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [23d8290f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:50:02,670-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [23d8290f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:50:02,672-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [417668fa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:50:02,673-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [417668fa] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:50:02,674-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [23d8290f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:50:02,675-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [23d8290f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:50:02,677-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [417668fa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:50:02,678-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [417668fa] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:50:02,678-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [23d8290f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:50:02,681-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [417668fa] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:50:02,681-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [23d8290f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 23d8290f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:50:02,688-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [417668fa] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 417668fa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:50:02,688-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4b88a77a
>2018-05-29 14:50:02,688-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:50:02,785-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:50:02,785-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:50:02,785-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:50:02,785-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:50:02,785-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4b88a77a
>2018-05-29 14:50:02,785-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:50:02,787-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 734f85f1
>2018-05-29 14:50:02,787-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:50:02,800-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [16c2eb38] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:50:02,801-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [16c2eb38] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 733d3244
>2018-05-29 14:50:02,803-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [16c2eb38] FINISH, SetVdsStatusVDSCommand, log id: 733d3244
>2018-05-29 14:50:02,835-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16c2eb38] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 16c2eb38, Job ID: 09037de6-2c64-4494-bbe7-e22dbc869e2c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:50:02,840-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16c2eb38] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 60c4d016, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:50:02,842-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16c2eb38] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 60c4d016, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:50:02,853-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [71ea243b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:50:02,854-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [71ea243b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:50:02,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:50:02,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:50:02,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:50:02,886-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:50:02,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 734f85f1
>2018-05-29 14:50:02,886-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:50:02,899-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2ede377c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:50:02,900-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2ede377c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1deaa36a
>2018-05-29 14:50:02,903-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2ede377c] FINISH, SetVdsStatusVDSCommand, log id: 1deaa36a
>2018-05-29 14:50:02,908-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2ede377c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2ede377c, Job ID: 39459e77-4b94-415a-adad-93d6e75988aa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:50:02,913-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2ede377c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 43852e18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:50:02,915-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2ede377c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 43852e18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:50:02,926-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [5fa486d9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:50:02,927-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [5fa486d9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:54:43,043-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler1) [2cb20409] Attempting to update VMs/Templates Ovf.
>2018-05-29 14:54:43,046-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 14:54:43,046-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 14:54:43,047-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:54:43,106-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 14:54:43,109-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 14:54:43,109-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 14:54:43,110-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 14:54:43,110-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 14:54:43,110-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 14:54:43,111-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 14:55:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [71396ce1] Autorecovering 2 hosts
>2018-05-29 14:55:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [71396ce1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:55:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6ea5bae3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6ea5bae3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6ea5bae3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6ea5bae3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:55:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6ea5bae3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4165789c
>2018-05-29 14:55:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6ea5bae3] FINISH, SetVdsStatusVDSCommand, log id: 4165789c
>2018-05-29 14:55:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6ea5bae3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:55:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6ea5bae3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 14:55:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [6ea5bae3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:55:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [eef3d9c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:55:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [eef3d9c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 14:55:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [eef3d9c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:55:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [eef3d9c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:55:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [eef3d9c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5d3b8579
>2018-05-29 14:55:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [eef3d9c] FINISH, SetVdsStatusVDSCommand, log id: 5d3b8579
>2018-05-29 14:55:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [eef3d9c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 645bc25
>2018-05-29 14:55:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [eef3d9c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 645bc25
>2018-05-29 14:55:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [eef3d9c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 14:55:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [eef3d9c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 14:55:00,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [269cb8e0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 29721de9
>2018-05-29 14:55:00,523-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [118c0464] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6ec45c6e
>2018-05-29 14:55:00,523-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [269cb8e0] FINISH, GetHardwareInfoVDSCommand, log id: 29721de9
>2018-05-29 14:55:00,525-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [118c0464] FINISH, GetHardwareInfoVDSCommand, log id: 6ec45c6e
>2018-05-29 14:55:00,537-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [43aa17df] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:55:00,540-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [71671723] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:55:00,546-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [505f10d9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:55:00,549-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [5709e504] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:55:00,551-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7ef89b88] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:55:00,554-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2946f922] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 14:55:00,563-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7ef89b88] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 693bcb36
>2018-05-29 14:55:00,566-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2946f922] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 307384c1
>2018-05-29 14:55:00,946-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7ef89b88] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:55:00,946-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7ef89b88] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 693bcb36
>2018-05-29 14:55:00,950-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7ef89b88] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:55:00,951-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7ef89b88] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:55:00,954-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7ef89b88] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:55:00,955-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7ef89b88] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:55:00,959-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7ef89b88] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:55:00,960-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7ef89b88] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:55:00,962-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7ef89b88] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:55:00,965-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7ef89b88] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7ef89b88, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:55:00,972-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 55120e80
>2018-05-29 14:55:00,972-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:55:01,070-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-11) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:55:01,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:55:01,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:55:01,070-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:55:01,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] FINISH, ConnectStoragePoolVDSCommand, log id: 55120e80
>2018-05-29 14:55:01,070-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-11) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:55:01,082-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [1c1f8fd9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:55:01,083-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1c1f8fd9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47f37f5b
>2018-05-29 14:55:01,085-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1c1f8fd9] FINISH, SetVdsStatusVDSCommand, log id: 47f37f5b
>2018-05-29 14:55:01,117-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1c1f8fd9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1c1f8fd9, Job ID: 1f7ec829-49a4-418d-a0da-83b66eb36891, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:55:01,121-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1c1f8fd9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 505f10d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:55:01,123-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1c1f8fd9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 505f10d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:55:01,134-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [3dc3f7b6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 14:55:01,135-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [3dc3f7b6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 14:55:01,932-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2946f922] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 14:55:01,932-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2946f922] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 307384c1
>2018-05-29 14:55:01,936-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2946f922] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:55:01,937-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2946f922] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:55:01,941-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2946f922] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:55:01,942-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2946f922] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:55:01,946-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2946f922] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 14:55:01,946-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2946f922] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 14:55:01,950-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2946f922] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 14:55:01,953-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2946f922] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2946f922, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 14:55:01,959-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 59080772
>2018-05-29 14:55:01,959-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 14:55:02,057-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:55:02,057-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 14:55:02,057-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 14:55:02,057-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 14:55:02,057-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 59080772
>2018-05-29 14:55:02,057-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 14:55:02,070-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [7dc5b050] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:55:02,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7dc5b050] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5724f032
>2018-05-29 14:55:02,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7dc5b050] FINISH, SetVdsStatusVDSCommand, log id: 5724f032
>2018-05-29 14:55:02,078-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7dc5b050] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7dc5b050, Job ID: be4c2e87-0852-460f-a03a-2b28f68ad6a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 14:55:02,083-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7dc5b050] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5709e504, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 14:55:02,085-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7dc5b050] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5709e504, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 14:55:02,095-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [39293cf6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 14:55:02,096-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [39293cf6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3575d8de] Autorecovering 2 hosts
>2018-05-29 15:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3575d8de] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:00:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [729b10f8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:00:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [729b10f8] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:00:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [729b10f8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:00:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [729b10f8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:00:00,055-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [729b10f8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4cb39fcd
>2018-05-29 15:00:00,057-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [729b10f8] FINISH, SetVdsStatusVDSCommand, log id: 4cb39fcd
>2018-05-29 15:00:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [729b10f8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:00:00,063-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [729b10f8] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 729b10f8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-29 15:00:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [729b10f8] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:00:00,064-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [729b10f8] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:00:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4def288b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4def288b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:00:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4def288b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:00:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4def288b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:00:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4def288b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43404a77
>2018-05-29 15:00:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4def288b] FINISH, SetVdsStatusVDSCommand, log id: 43404a77
>2018-05-29 15:00:00,082-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4def288b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4fd72b39
>2018-05-29 15:00:00,084-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4def288b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4fd72b39
>2018-05-29 15:00:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4def288b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:00:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4def288b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:00:01,776-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6ad8cc7c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 12dde6c4
>2018-05-29 15:00:01,778-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6ad8cc7c] FINISH, GetHardwareInfoVDSCommand, log id: 12dde6c4
>2018-05-29 15:00:01,792-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [48e8a1d9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:00:01,800-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [6f5c6bd4] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:00:01,805-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [60c8441c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:00:01,811-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [60c8441c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e1b6425
>2018-05-29 15:00:02,197-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60c8441c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:00:02,197-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [60c8441c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e1b6425
>2018-05-29 15:00:02,201-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60c8441c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:00:02,201-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [60c8441c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:00:02,205-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60c8441c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:00:02,206-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [60c8441c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:00:02,210-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60c8441c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:00:02,210-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [60c8441c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:00:02,214-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [60c8441c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:00:02,217-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60c8441c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 60c8441c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:00:02,220-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1bf39bf0
>2018-05-29 15:00:02,220-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:00:02,316-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:00:02,316-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:00:02,316-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:00:02,316-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:00:02,316-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1bf39bf0
>2018-05-29 15:00:02,317-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:00:02,330-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [406ee23] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:00:02,331-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [406ee23] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3a1404e
>2018-05-29 15:00:02,333-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [406ee23] FINISH, SetVdsStatusVDSCommand, log id: 3a1404e
>2018-05-29 15:00:02,365-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [406ee23] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 406ee23, Job ID: 59ca2fa5-5540-4b02-84cd-1ba8dd0643e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:00:02,369-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [406ee23] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6f5c6bd4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:00:02,371-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [406ee23] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6f5c6bd4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:00:02,382-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [34ddb122] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:00:02,383-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [34ddb122] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:00:02,752-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [8151837] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 64dcd54f
>2018-05-29 15:00:02,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [8151837] FINISH, GetHardwareInfoVDSCommand, log id: 64dcd54f
>2018-05-29 15:00:02,766-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [6acf473e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:00:02,774-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [69b739c8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:00:02,779-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [694be675] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:00:02,786-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [694be675] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6ea55e5d
>2018-05-29 15:00:03,156-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [694be675] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:00:03,156-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [694be675] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6ea55e5d
>2018-05-29 15:00:03,160-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [694be675] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:00:03,160-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [694be675] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:00:03,164-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [694be675] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:00:03,165-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [694be675] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:00:03,169-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [694be675] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:00:03,170-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [694be675] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:00:03,172-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [694be675] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:00:03,175-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [694be675] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 694be675, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:00:03,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 63a532d9
>2018-05-29 15:00:03,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:00:03,277-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:00:03,277-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:00:03,277-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:00:03,277-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:00:03,277-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 63a532d9
>2018-05-29 15:00:03,277-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:00:03,290-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [43c94b41] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:00:03,291-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [43c94b41] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5313dae8
>2018-05-29 15:00:03,293-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [43c94b41] FINISH, SetVdsStatusVDSCommand, log id: 5313dae8
>2018-05-29 15:00:03,298-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [43c94b41] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 43c94b41, Job ID: f1a1811b-057f-4de0-b47d-4c0d9fc2ecbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:00:03,303-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [43c94b41] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 69b739c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:00:03,305-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [43c94b41] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 69b739c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:00:03,315-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [3ae845f0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:00:03,316-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [3ae845f0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [14107945] Autorecovering 2 hosts
>2018-05-29 15:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [14107945] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:05:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d7ec314] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:05:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d7ec314] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:05:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d7ec314] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:05:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d7ec314] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:05:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7d7ec314] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 275e13d4
>2018-05-29 15:05:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7d7ec314] FINISH, SetVdsStatusVDSCommand, log id: 275e13d4
>2018-05-29 15:05:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d7ec314] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:05:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d7ec314] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:05:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7d7ec314] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:05:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17c20070] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:05:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17c20070] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:05:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17c20070] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:05:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17c20070] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:05:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [17c20070] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52e69fea
>2018-05-29 15:05:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [17c20070] FINISH, SetVdsStatusVDSCommand, log id: 52e69fea
>2018-05-29 15:05:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [17c20070] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7acbf263
>2018-05-29 15:05:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [17c20070] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7acbf263
>2018-05-29 15:05:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17c20070] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:05:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17c20070] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:05:00,970-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [287b1a54] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4fadc395
>2018-05-29 15:05:00,972-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [287b1a54] FINISH, GetHardwareInfoVDSCommand, log id: 4fadc395
>2018-05-29 15:05:00,985-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [538da7d5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:05:00,990-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [1d8496a8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:05:00,995-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5414f21b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:05:01,006-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5414f21b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1ac37d51
>2018-05-29 15:05:02,379-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5414f21b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:05:02,379-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5414f21b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1ac37d51
>2018-05-29 15:05:02,383-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5414f21b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:05:02,384-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5414f21b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:05:02,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5414f21b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:05:02,388-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5414f21b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:05:02,392-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5414f21b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:05:02,393-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5414f21b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:05:02,439-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5414f21b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:05:02,443-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5414f21b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5414f21b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:05:02,449-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 37f49a61
>2018-05-29 15:05:02,450-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:05:02,548-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:05:02,548-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:05:02,548-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:05:02,548-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:05:02,548-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 37f49a61
>2018-05-29 15:05:02,548-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:05:02,562-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2d3ee45d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:05:02,564-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2d3ee45d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5b68f3a3
>2018-05-29 15:05:02,566-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2d3ee45d] FINISH, SetVdsStatusVDSCommand, log id: 5b68f3a3
>2018-05-29 15:05:02,597-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2d3ee45d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2d3ee45d, Job ID: e8cb9ec0-6385-45c5-b429-1d4ad90324df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:05:02,602-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2d3ee45d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d8496a8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:05:02,604-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2d3ee45d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d8496a8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:05:02,615-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [68179374] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:05:02,616-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [68179374] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:05:02,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2fc9e613] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 55a73ded
>2018-05-29 15:05:02,999-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2fc9e613] FINISH, GetHardwareInfoVDSCommand, log id: 55a73ded
>2018-05-29 15:05:03,011-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [74a64b0a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:05:03,019-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [4620e0da] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:05:03,024-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5461dac3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:05:03,035-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5461dac3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22f1c110
>2018-05-29 15:05:03,421-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5461dac3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:05:03,421-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5461dac3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 22f1c110
>2018-05-29 15:05:03,424-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5461dac3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:05:03,425-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5461dac3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:05:03,429-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5461dac3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:05:03,430-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5461dac3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:05:03,433-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5461dac3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:05:03,434-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5461dac3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:05:03,437-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5461dac3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:05:03,439-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5461dac3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5461dac3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:05:03,442-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5b3b84df
>2018-05-29 15:05:03,442-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:05:03,542-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:05:03,542-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:05:03,542-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:05:03,542-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:05:03,542-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5b3b84df
>2018-05-29 15:05:03,542-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:05:03,555-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [6919528] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:05:03,556-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6919528] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 550afee9
>2018-05-29 15:05:03,558-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6919528] FINISH, SetVdsStatusVDSCommand, log id: 550afee9
>2018-05-29 15:05:03,564-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6919528] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6919528, Job ID: 036df527-fae7-4428-99bf-6e1c7cde5206, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:05:03,568-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6919528] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4620e0da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:05:03,570-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6919528] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4620e0da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:05:03,580-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [109ef1a8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:05:03,582-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [109ef1a8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [32ff4741] Autorecovering 2 hosts
>2018-05-29 15:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [32ff4741] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:10:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6209675] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:10:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6209675] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:10:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6209675] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:10:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6209675] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:10:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6209675] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c9a8fd7
>2018-05-29 15:10:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6209675] FINISH, SetVdsStatusVDSCommand, log id: 2c9a8fd7
>2018-05-29 15:10:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6209675] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:10:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6209675] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:10:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [6209675] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:10:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3f6f4e29] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:10:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3f6f4e29] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:10:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3f6f4e29] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:10:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3f6f4e29] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:10:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3f6f4e29] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b97b145
>2018-05-29 15:10:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3f6f4e29] FINISH, SetVdsStatusVDSCommand, log id: 4b97b145
>2018-05-29 15:10:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [3f6f4e29] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 52532b1d
>2018-05-29 15:10:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [3f6f4e29] FINISH, SetHaMaintenanceModeVDSCommand, log id: 52532b1d
>2018-05-29 15:10:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3f6f4e29] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:10:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3f6f4e29] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:10:00,262-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [575a8689] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7f85a9c3
>2018-05-29 15:10:00,263-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [575a8689] FINISH, GetHardwareInfoVDSCommand, log id: 7f85a9c3
>2018-05-29 15:10:00,276-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [506a27d7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:10:00,284-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [64a45fc5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:10:00,290-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [14554780] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:10:00,301-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [14554780] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2fe2baa2
>2018-05-29 15:10:00,673-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14554780] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:10:00,673-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [14554780] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2fe2baa2
>2018-05-29 15:10:00,677-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14554780] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:10:00,678-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [14554780] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:10:00,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14554780] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:10:00,682-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [14554780] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:10:00,686-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14554780] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:10:00,687-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [14554780] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:10:00,690-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [14554780] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:10:00,693-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14554780] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 14554780, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:10:00,701-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5cf7a6ac
>2018-05-29 15:10:00,701-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:10:00,802-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:10:00,802-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:10:00,802-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:10:00,802-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:10:00,802-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5cf7a6ac
>2018-05-29 15:10:00,802-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:10:00,817-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [5daeeebd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:10:00,818-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5daeeebd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 576669d1
>2018-05-29 15:10:00,820-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5daeeebd] FINISH, SetVdsStatusVDSCommand, log id: 576669d1
>2018-05-29 15:10:00,855-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5daeeebd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5daeeebd, Job ID: 317d7c43-7c90-4398-bdcf-0728aef211da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:10:00,860-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5daeeebd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 64a45fc5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:10:00,862-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5daeeebd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 64a45fc5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:10:00,874-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [33e18823] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:10:00,875-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [33e18823] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:10:01,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [203ec591] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 763443f2
>2018-05-29 15:10:01,282-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [203ec591] FINISH, GetHardwareInfoVDSCommand, log id: 763443f2
>2018-05-29 15:10:01,295-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [66b3e822] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:10:01,303-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [55599a69] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:10:01,309-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [77e0e876] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:10:01,321-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [77e0e876] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 223c4c50
>2018-05-29 15:10:01,707-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [77e0e876] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:10:01,707-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [77e0e876] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 223c4c50
>2018-05-29 15:10:01,711-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [77e0e876] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:10:01,711-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [77e0e876] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:10:01,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [77e0e876] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:10:01,716-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [77e0e876] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:10:01,720-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [77e0e876] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:10:01,721-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [77e0e876] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:10:01,723-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [77e0e876] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:10:01,726-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [77e0e876] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 77e0e876, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:10:01,732-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2e0747b5
>2018-05-29 15:10:01,732-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:10:01,829-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:10:01,829-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:10:01,829-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:10:01,829-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:10:01,829-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2e0747b5
>2018-05-29 15:10:01,829-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:10:01,843-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [3d01fb4d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:10:01,844-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3d01fb4d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6c90bc80
>2018-05-29 15:10:01,846-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3d01fb4d] FINISH, SetVdsStatusVDSCommand, log id: 6c90bc80
>2018-05-29 15:10:01,877-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d01fb4d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3d01fb4d, Job ID: 49af8d0b-b7e8-40a0-9346-0419f9d69226, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:10:01,882-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d01fb4d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 55599a69, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:10:01,885-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d01fb4d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 55599a69, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:10:01,896-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [1bd30273] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:10:01,897-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [1bd30273] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [74531bf5] Autorecovering 2 hosts
>2018-05-29 15:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [74531bf5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:15:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ac46f83] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:15:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ac46f83] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:15:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ac46f83] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:15:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ac46f83] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:15:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3ac46f83] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52e74c40
>2018-05-29 15:15:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3ac46f83] FINISH, SetVdsStatusVDSCommand, log id: 52e74c40
>2018-05-29 15:15:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ac46f83] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:15:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ac46f83] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:15:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3ac46f83] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:15:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b869bf6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:15:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b869bf6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:15:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b869bf6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:15:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b869bf6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:15:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5b869bf6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f7aa801
>2018-05-29 15:15:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5b869bf6] FINISH, SetVdsStatusVDSCommand, log id: 1f7aa801
>2018-05-29 15:15:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [5b869bf6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 35798a1b
>2018-05-29 15:15:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [5b869bf6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 35798a1b
>2018-05-29 15:15:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b869bf6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:15:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b869bf6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:15:01,498-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [560e71c4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 8b24c29
>2018-05-29 15:15:01,500-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [560e71c4] FINISH, GetHardwareInfoVDSCommand, log id: 8b24c29
>2018-05-29 15:15:01,513-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [c12d76a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:15:01,521-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [345273c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:15:01,526-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [396728c3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:15:01,538-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [396728c3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1d016147
>2018-05-29 15:15:01,906-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [396728c3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:15:01,906-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [396728c3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1d016147
>2018-05-29 15:15:01,910-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [396728c3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:15:01,911-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [396728c3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:15:01,915-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [396728c3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:15:01,916-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [396728c3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:15:01,919-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [396728c3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:15:01,920-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [396728c3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:15:01,923-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [396728c3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:15:01,925-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [396728c3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 396728c3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:15:01,933-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 182b0fe
>2018-05-29 15:15:01,933-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:15:02,030-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-11) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:15:02,030-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:15:02,030-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:15:02,030-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:15:02,030-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] FINISH, ConnectStoragePoolVDSCommand, log id: 182b0fe
>2018-05-29 15:15:02,030-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-11) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:15:02,045-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [4fd1c8f2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:15:02,046-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4fd1c8f2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 606e1597
>2018-05-29 15:15:02,048-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4fd1c8f2] FINISH, SetVdsStatusVDSCommand, log id: 606e1597
>2018-05-29 15:15:02,084-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4fd1c8f2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4fd1c8f2, Job ID: 265af355-c98f-442b-b5aa-4a4bccaa302c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:15:02,088-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4fd1c8f2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 345273c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:15:02,091-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4fd1c8f2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 345273c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:15:02,104-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [3487ac01] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:15:02,105-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [3487ac01] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:15:02,582-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [221d6cae] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 78af985f
>2018-05-29 15:15:02,584-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [221d6cae] FINISH, GetHardwareInfoVDSCommand, log id: 78af985f
>2018-05-29 15:15:02,597-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [622c99b8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:15:02,602-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [c44d32] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:15:02,607-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5bf160e2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:15:02,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5bf160e2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 473a8619
>2018-05-29 15:15:02,993-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bf160e2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:15:02,993-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5bf160e2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 473a8619
>2018-05-29 15:15:02,998-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bf160e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:15:02,998-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5bf160e2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:15:03,002-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bf160e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:15:03,003-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5bf160e2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:15:03,007-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bf160e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:15:03,008-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5bf160e2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:15:03,010-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5bf160e2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:15:03,013-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bf160e2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5bf160e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:15:03,016-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 568fe6c7
>2018-05-29 15:15:03,016-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:15:03,112-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:15:03,112-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:15:03,112-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:15:03,112-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:15:03,112-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 568fe6c7
>2018-05-29 15:15:03,112-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:15:03,126-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [308d94bb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:15:03,127-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [308d94bb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3e30f8cf
>2018-05-29 15:15:03,129-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [308d94bb] FINISH, SetVdsStatusVDSCommand, log id: 3e30f8cf
>2018-05-29 15:15:03,158-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [308d94bb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 308d94bb, Job ID: 5131c316-1f23-4db5-9ba8-176654dd60f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:15:03,163-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [308d94bb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: c44d32, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:15:03,165-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [308d94bb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: c44d32, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:15:03,176-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [b48b976] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:15:03,177-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [b48b976] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:20:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [33587eef] Autorecovering 2 hosts
>2018-05-29 15:20:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [33587eef] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:20:00,033-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4e7f765c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:20:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4e7f765c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:20:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4e7f765c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:20:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4e7f765c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:20:00,041-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4e7f765c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53025302
>2018-05-29 15:20:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4e7f765c] FINISH, SetVdsStatusVDSCommand, log id: 53025302
>2018-05-29 15:20:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4e7f765c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:20:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4e7f765c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:20:00,047-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [4e7f765c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:20:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [31c3f99d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:20:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [31c3f99d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:20:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [31c3f99d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:20:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [31c3f99d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:20:00,060-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [31c3f99d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6bb97728
>2018-05-29 15:20:00,062-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [31c3f99d] FINISH, SetVdsStatusVDSCommand, log id: 6bb97728
>2018-05-29 15:20:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [31c3f99d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 153c4644
>2018-05-29 15:20:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [31c3f99d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 153c4644
>2018-05-29 15:20:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [31c3f99d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:20:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [31c3f99d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:20:00,779-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7ec2e7bb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7e61c37
>2018-05-29 15:20:00,781-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7ec2e7bb] FINISH, GetHardwareInfoVDSCommand, log id: 7e61c37
>2018-05-29 15:20:00,794-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [47b7b88a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:20:00,799-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [5f0572bc] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:20:00,804-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7fbe17ba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:20:00,861-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7fbe17ba] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3b79e3e5
>2018-05-29 15:20:01,239-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fbe17ba] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:20:01,239-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7fbe17ba] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3b79e3e5
>2018-05-29 15:20:01,244-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fbe17ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:20:01,244-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7fbe17ba] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:20:01,248-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fbe17ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:20:01,249-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7fbe17ba] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:20:01,253-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fbe17ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:20:01,253-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7fbe17ba] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:20:01,256-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7fbe17ba] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:20:01,259-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fbe17ba] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7fbe17ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:20:01,265-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1f344710
>2018-05-29 15:20:01,265-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:20:01,361-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:20:01,361-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:20:01,361-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:20:01,361-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:20:01,361-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1f344710
>2018-05-29 15:20:01,361-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:20:01,374-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [73da2e13] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:20:01,375-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [73da2e13] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 66042d49
>2018-05-29 15:20:01,377-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [73da2e13] FINISH, SetVdsStatusVDSCommand, log id: 66042d49
>2018-05-29 15:20:01,386-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [73da2e13] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 73da2e13, Job ID: 293ed708-ce5a-4dca-8382-3e1c813d4eae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:20:01,390-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [73da2e13] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5f0572bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:20:01,393-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [73da2e13] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5f0572bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:20:01,404-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [4a9d899c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:20:01,405-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [4a9d899c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:20:02,895-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [55429984] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6d9d7c0f
>2018-05-29 15:20:02,896-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [55429984] FINISH, GetHardwareInfoVDSCommand, log id: 6d9d7c0f
>2018-05-29 15:20:02,937-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [247bba3a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:20:02,942-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [1ae11507] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:20:02,948-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1e70bb72] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:20:02,959-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1e70bb72] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 389fe0c3
>2018-05-29 15:20:03,337-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e70bb72] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:20:03,337-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1e70bb72] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 389fe0c3
>2018-05-29 15:20:03,341-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e70bb72] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:20:03,341-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e70bb72] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:20:03,345-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e70bb72] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:20:03,346-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e70bb72] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:20:03,350-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e70bb72] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:20:03,351-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e70bb72] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:20:03,353-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1e70bb72] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:20:03,356-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e70bb72] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1e70bb72, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:20:03,363-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 217b7c59
>2018-05-29 15:20:03,363-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:20:03,463-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:20:03,463-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:20:03,463-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:20:03,463-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:20:03,463-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 217b7c59
>2018-05-29 15:20:03,463-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:20:03,477-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [630c8905] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:20:03,478-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [630c8905] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6e9ce2c1
>2018-05-29 15:20:03,481-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [630c8905] FINISH, SetVdsStatusVDSCommand, log id: 6e9ce2c1
>2018-05-29 15:20:03,515-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [630c8905] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 630c8905, Job ID: 17754910-29c1-4cdd-910b-8c6a1b32752a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:20:03,519-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [630c8905] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1ae11507, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:20:03,522-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [630c8905] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1ae11507, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:20:03,532-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [3048f7cf] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:20:03,533-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [3048f7cf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:25:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4b8b59eb] Autorecovering 2 hosts
>2018-05-29 15:25:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4b8b59eb] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:25:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5d659161] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:25:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5d659161] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:25:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5d659161] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:25:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5d659161] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:25:00,053-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5d659161] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 699bc4f2
>2018-05-29 15:25:00,056-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5d659161] FINISH, SetVdsStatusVDSCommand, log id: 699bc4f2
>2018-05-29 15:25:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5d659161] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:25:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5d659161] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:25:00,059-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5d659161] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:25:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6b3f2975] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:25:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6b3f2975] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:25:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6b3f2975] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:25:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6b3f2975] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:25:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6b3f2975] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 225b29f9
>2018-05-29 15:25:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6b3f2975] FINISH, SetVdsStatusVDSCommand, log id: 225b29f9
>2018-05-29 15:25:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6b3f2975] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7a12e1d
>2018-05-29 15:25:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6b3f2975] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7a12e1d
>2018-05-29 15:25:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6b3f2975] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:25:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6b3f2975] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:25:01,274-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [16d3ab9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 58e87b1e
>2018-05-29 15:25:01,276-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [16d3ab9] FINISH, GetHardwareInfoVDSCommand, log id: 58e87b1e
>2018-05-29 15:25:01,290-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [5bd9180d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:25:01,299-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [4f31a2c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:25:01,304-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [58966f89] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:25:01,310-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [58966f89] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3ccdbe64
>2018-05-29 15:25:02,258-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [535f3c90] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 11f55a8f
>2018-05-29 15:25:02,260-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [535f3c90] FINISH, GetHardwareInfoVDSCommand, log id: 11f55a8f
>2018-05-29 15:25:02,280-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [10e9107e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:25:02,289-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [638b8bb1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:25:02,294-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2f986542] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:25:02,301-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2f986542] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 12ca8ba9
>2018-05-29 15:25:02,674-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58966f89] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:25:02,674-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [58966f89] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3ccdbe64
>2018-05-29 15:25:02,678-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58966f89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:25:02,679-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [58966f89] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:25:02,681-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f986542] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:25:02,681-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2f986542] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 12ca8ba9
>2018-05-29 15:25:02,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58966f89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:25:02,683-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [58966f89] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:25:02,686-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f986542] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:25:02,686-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2f986542] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:25:02,687-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58966f89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:25:02,688-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [58966f89] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:25:02,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f986542] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:25:02,691-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [58966f89] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:25:02,691-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2f986542] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:25:02,694-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58966f89] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 58966f89, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:25:02,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f986542] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:25:02,696-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2f986542] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:25:02,696-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2796478e
>2018-05-29 15:25:02,696-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:25:02,700-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2f986542] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:25:02,702-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f986542] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2f986542, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:25:02,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:25:02,792-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:25:02,792-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:25:02,792-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:25:02,792-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2796478e
>2018-05-29 15:25:02,792-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:25:02,799-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 629b08e0
>2018-05-29 15:25:02,799-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:25:02,806-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5069bfd8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:25:02,807-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5069bfd8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32f1addc
>2018-05-29 15:25:02,809-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5069bfd8] FINISH, SetVdsStatusVDSCommand, log id: 32f1addc
>2018-05-29 15:25:02,838-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5069bfd8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5069bfd8, Job ID: 0de22425-82fa-49a9-b3dd-2301a8ddf6c4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:25:02,842-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5069bfd8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4f31a2c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:25:02,845-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5069bfd8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4f31a2c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:25:02,856-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [66b54643] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:25:02,857-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [66b54643] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:25:02,898-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:25:02,899-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:25:02,899-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:25:02,899-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:25:02,899-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 629b08e0
>2018-05-29 15:25:02,899-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:25:02,912-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [40d80186] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:25:02,913-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [40d80186] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 574c3a88
>2018-05-29 15:25:02,915-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [40d80186] FINISH, SetVdsStatusVDSCommand, log id: 574c3a88
>2018-05-29 15:25:02,921-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [40d80186] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 40d80186, Job ID: 0f3ba775-d7e1-43a8-863e-c305431b9eab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:25:02,925-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [40d80186] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 638b8bb1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:25:02,927-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [40d80186] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 638b8bb1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:25:02,938-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [545f9812] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:25:02,939-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [545f9812] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:30:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4f5ce181] Autorecovering 2 hosts
>2018-05-29 15:30:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4f5ce181] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:30:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [292c4ecf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:30:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [292c4ecf] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:30:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [292c4ecf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:30:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [292c4ecf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:30:00,049-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [292c4ecf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: fc20e2
>2018-05-29 15:30:00,052-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [292c4ecf] FINISH, SetVdsStatusVDSCommand, log id: fc20e2
>2018-05-29 15:30:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [292c4ecf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:30:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [292c4ecf] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:30:00,055-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [292c4ecf] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:30:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6c681466] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:30:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6c681466] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:30:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6c681466] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:30:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6c681466] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:30:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6c681466] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49a79bb3
>2018-05-29 15:30:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6c681466] FINISH, SetVdsStatusVDSCommand, log id: 49a79bb3
>2018-05-29 15:30:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6c681466] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 342a1c6
>2018-05-29 15:30:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6c681466] FINISH, SetHaMaintenanceModeVDSCommand, log id: 342a1c6
>2018-05-29 15:30:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6c681466] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:30:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6c681466] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:30:00,499-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [66bbe354] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 46274c6b
>2018-05-29 15:30:00,501-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [66bbe354] FINISH, GetHardwareInfoVDSCommand, log id: 46274c6b
>2018-05-29 15:30:00,514-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [21c9828a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:30:00,523-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [125e3d17] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:30:00,528-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [646f0236] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:30:00,587-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [646f0236] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 53895a2e
>2018-05-29 15:30:00,592-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [294ff49] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7650b311
>2018-05-29 15:30:00,594-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [294ff49] FINISH, GetHardwareInfoVDSCommand, log id: 7650b311
>2018-05-29 15:30:00,607-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [23925a75] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:30:00,616-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [77dad3fc] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:30:00,621-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [23924d5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:30:00,632-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [23924d5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2c4d58e1
>2018-05-29 15:30:00,969-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [646f0236] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:30:00,970-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [646f0236] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 53895a2e
>2018-05-29 15:30:00,974-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [646f0236] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:30:00,975-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [646f0236] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:30:00,979-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [646f0236] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:30:00,979-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [646f0236] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:30:00,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [646f0236] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:30:00,984-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [646f0236] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:30:00,987-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [646f0236] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:30:00,990-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [646f0236] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 646f0236, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:30:00,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a35924b
>2018-05-29 15:30:00,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:30:01,005-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [23924d5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:30:01,005-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [23924d5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2c4d58e1
>2018-05-29 15:30:01,008-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [23924d5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:30:01,009-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [23924d5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:30:01,013-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [23924d5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:30:01,014-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [23924d5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:30:01,017-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [23924d5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:30:01,018-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [23924d5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:30:01,072-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [23924d5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:30:01,076-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [23924d5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 23924d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:30:01,095-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:30:01,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:30:01,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:30:01,095-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:30:01,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a35924b
>2018-05-29 15:30:01,095-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:30:01,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6babc3b6
>2018-05-29 15:30:01,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:30:01,111-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [7095ad8a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:30:01,112-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7095ad8a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 585123b5
>2018-05-29 15:30:01,114-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7095ad8a] FINISH, SetVdsStatusVDSCommand, log id: 585123b5
>2018-05-29 15:30:01,123-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7095ad8a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7095ad8a, Job ID: c7ed451d-29d3-48c5-8381-3d46de42a0b1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:30:01,128-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7095ad8a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 125e3d17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:30:01,130-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7095ad8a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 125e3d17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:30:01,141-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [59599534] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:30:01,142-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [59599534] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:30:01,200-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:30:01,200-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:30:01,200-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:30:01,200-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:30:01,200-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6babc3b6
>2018-05-29 15:30:01,200-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:30:01,213-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [7c5e42de] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:30:01,214-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7c5e42de] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 173f208a
>2018-05-29 15:30:01,216-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7c5e42de] FINISH, SetVdsStatusVDSCommand, log id: 173f208a
>2018-05-29 15:30:01,222-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7c5e42de] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7c5e42de, Job ID: c7f83e4b-f271-4106-b31d-41e7b42ad124, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:30:01,227-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7c5e42de] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 77dad3fc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:30:01,229-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7c5e42de] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 77dad3fc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:30:01,239-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [502f731a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:30:01,240-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [502f731a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7328dbdf] Autorecovering 2 hosts
>2018-05-29 15:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7328dbdf] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:35:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [542685e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:35:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [542685e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:35:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [542685e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:35:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [542685e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:35:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [542685e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 65eb0110
>2018-05-29 15:35:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [542685e] FINISH, SetVdsStatusVDSCommand, log id: 65eb0110
>2018-05-29 15:35:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [542685e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:35:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [542685e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:35:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [542685e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:35:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [562f6d4f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:35:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [562f6d4f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:35:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [562f6d4f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:35:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [562f6d4f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:35:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [562f6d4f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7cf7bc29
>2018-05-29 15:35:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [562f6d4f] FINISH, SetVdsStatusVDSCommand, log id: 7cf7bc29
>2018-05-29 15:35:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [562f6d4f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 43ceb93d
>2018-05-29 15:35:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [562f6d4f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 43ceb93d
>2018-05-29 15:35:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [562f6d4f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:35:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [562f6d4f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:35:01,805-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3da14bea] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 192891c5
>2018-05-29 15:35:01,806-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3da14bea] FINISH, GetHardwareInfoVDSCommand, log id: 192891c5
>2018-05-29 15:35:01,821-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [436a3845] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:35:01,830-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [5bba2c14] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:35:01,836-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [bfea202] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:35:01,848-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [bfea202] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 261d3fdf
>2018-05-29 15:35:01,964-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2032a8fc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 13955862
>2018-05-29 15:35:01,966-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2032a8fc] FINISH, GetHardwareInfoVDSCommand, log id: 13955862
>2018-05-29 15:35:01,980-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [52f6cc78] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:35:01,984-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [652929ba] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:35:01,989-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [21a99b50] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:35:01,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [21a99b50] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: e581bc3
>2018-05-29 15:35:02,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [bfea202] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:35:02,226-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [bfea202] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 261d3fdf
>2018-05-29 15:35:02,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [bfea202] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:35:02,231-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [bfea202] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:35:02,235-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [bfea202] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:35:02,236-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [bfea202] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:35:02,239-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [bfea202] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:35:02,240-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [bfea202] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:35:02,243-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [bfea202] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:35:02,245-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [bfea202] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: bfea202, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:35:02,252-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7597470f
>2018-05-29 15:35:02,252-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:35:02,346-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:35:02,346-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:35:02,346-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:35:02,346-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:35:02,346-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7597470f
>2018-05-29 15:35:02,346-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:35:02,360-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4d5f0ae4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:35:02,361-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4d5f0ae4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 64d5aced
>2018-05-29 15:35:02,363-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4d5f0ae4] FINISH, SetVdsStatusVDSCommand, log id: 64d5aced
>2018-05-29 15:35:02,395-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4d5f0ae4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4d5f0ae4, Job ID: 917e04a1-d919-4e9a-b076-6179a867d971, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:35:02,400-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4d5f0ae4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5bba2c14, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:35:02,402-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4d5f0ae4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5bba2c14, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:35:02,414-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [6d087502] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:35:02,415-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [6d087502] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:35:03,369-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21a99b50] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:35:03,369-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [21a99b50] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: e581bc3
>2018-05-29 15:35:03,373-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21a99b50] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:35:03,374-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [21a99b50] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:35:03,378-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21a99b50] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:35:03,379-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [21a99b50] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:35:03,383-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21a99b50] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:35:03,383-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [21a99b50] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:35:03,387-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [21a99b50] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:35:03,390-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21a99b50] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 21a99b50, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:35:03,393-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5dc56b7c
>2018-05-29 15:35:03,393-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:35:03,492-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:35:03,492-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:35:03,492-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:35:03,492-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:35:03,492-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5dc56b7c
>2018-05-29 15:35:03,492-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:35:03,506-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [10176208] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:35:03,507-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [10176208] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5699b4e0
>2018-05-29 15:35:03,509-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [10176208] FINISH, SetVdsStatusVDSCommand, log id: 5699b4e0
>2018-05-29 15:35:03,542-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [10176208] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 10176208, Job ID: a7489940-b382-453a-8e8e-e86e3b2b15a9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:35:03,547-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [10176208] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 652929ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:35:03,549-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [10176208] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 652929ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:35:03,560-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [4fcf05b9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:35:03,561-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [4fcf05b9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:40:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [45d652ae] Autorecovering 2 hosts
>2018-05-29 15:40:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [45d652ae] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:40:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [393cab7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:40:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [393cab7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:40:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [393cab7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:40:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [393cab7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:40:00,053-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [393cab7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1d80c
>2018-05-29 15:40:00,056-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [393cab7] FINISH, SetVdsStatusVDSCommand, log id: 1d80c
>2018-05-29 15:40:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [393cab7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:40:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [393cab7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:40:00,059-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [393cab7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:40:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6a0907ca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6a0907ca] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6a0907ca] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6a0907ca] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:40:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6a0907ca] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 657bec
>2018-05-29 15:40:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6a0907ca] FINISH, SetVdsStatusVDSCommand, log id: 657bec
>2018-05-29 15:40:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [6a0907ca] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 9b66ec4
>2018-05-29 15:40:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [6a0907ca] FINISH, SetHaMaintenanceModeVDSCommand, log id: 9b66ec4
>2018-05-29 15:40:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6a0907ca] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:40:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6a0907ca] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:40:01,317-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5140f9c2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2de495e9
>2018-05-29 15:40:01,319-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5140f9c2] FINISH, GetHardwareInfoVDSCommand, log id: 2de495e9
>2018-05-29 15:40:01,333-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [51e46d0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:40:01,341-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [227bfab] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:40:01,347-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [68ac57e1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:40:01,353-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [68ac57e1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: eeccae4
>2018-05-29 15:40:01,720-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68ac57e1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:40:01,720-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [68ac57e1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: eeccae4
>2018-05-29 15:40:01,724-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68ac57e1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:40:01,725-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [68ac57e1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:40:01,729-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68ac57e1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:40:01,729-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [68ac57e1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:40:01,733-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68ac57e1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:40:01,734-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [68ac57e1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:40:01,737-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [68ac57e1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:40:01,739-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68ac57e1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 68ac57e1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:40:01,742-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 37736488
>2018-05-29 15:40:01,743-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:40:01,843-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:40:01,843-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:40:01,843-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:40:01,843-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:40:01,843-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 37736488
>2018-05-29 15:40:01,843-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:40:01,856-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [2b5d1815] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:40:01,858-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2b5d1815] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6fff12a7
>2018-05-29 15:40:01,860-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2b5d1815] FINISH, SetVdsStatusVDSCommand, log id: 6fff12a7
>2018-05-29 15:40:01,889-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2b5d1815] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2b5d1815, Job ID: 21f55bb3-34ad-48f3-b999-717bbb438840, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:40:01,893-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2b5d1815] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 227bfab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:40:01,895-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2b5d1815] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 227bfab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:40:01,906-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [3d0be4b7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:40:01,907-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [3d0be4b7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:40:03,009-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [51499c04] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 152d9a1
>2018-05-29 15:40:03,011-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [51499c04] FINISH, GetHardwareInfoVDSCommand, log id: 152d9a1
>2018-05-29 15:40:03,024-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [3d9c6237] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:40:03,032-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2273bffd] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:40:03,037-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [52ddac85] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:40:03,049-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [52ddac85] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 295c3e67
>2018-05-29 15:40:03,430-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [52ddac85] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:40:03,430-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [52ddac85] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 295c3e67
>2018-05-29 15:40:03,434-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [52ddac85] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:40:03,435-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [52ddac85] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:40:03,438-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [52ddac85] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:40:03,439-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [52ddac85] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:40:03,443-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [52ddac85] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:40:03,444-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [52ddac85] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:40:03,446-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [52ddac85] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:40:03,449-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [52ddac85] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 52ddac85, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:40:03,455-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 9a10141
>2018-05-29 15:40:03,455-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:40:03,554-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:40:03,554-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:40:03,554-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:40:03,554-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:40:03,554-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 9a10141
>2018-05-29 15:40:03,554-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:40:03,567-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [45df9312] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:40:03,569-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [45df9312] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43b39f18
>2018-05-29 15:40:03,571-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [45df9312] FINISH, SetVdsStatusVDSCommand, log id: 43b39f18
>2018-05-29 15:40:03,603-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [45df9312] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 45df9312, Job ID: 6d29a7bc-bb36-41e0-8213-af8b98e29212, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:40:03,608-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [45df9312] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2273bffd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:40:03,610-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [45df9312] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2273bffd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:40:03,621-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [67474c72] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:40:03,622-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [67474c72] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [645b6db7] Autorecovering 2 hosts
>2018-05-29 15:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [645b6db7] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:45:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [434a458a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:45:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [434a458a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:45:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [434a458a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:45:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [434a458a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:45:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [434a458a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d947d2d
>2018-05-29 15:45:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [434a458a] FINISH, SetVdsStatusVDSCommand, log id: 3d947d2d
>2018-05-29 15:45:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [434a458a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:45:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [434a458a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:45:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [434a458a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:45:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [560220c3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:45:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [560220c3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:45:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [560220c3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:45:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [560220c3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:45:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [560220c3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 40bbf578
>2018-05-29 15:45:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [560220c3] FINISH, SetVdsStatusVDSCommand, log id: 40bbf578
>2018-05-29 15:45:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [560220c3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6e9a97e8
>2018-05-29 15:45:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [560220c3] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6e9a97e8
>2018-05-29 15:45:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [560220c3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:45:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [560220c3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:45:01,224-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [49723e73] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 726c922f
>2018-05-29 15:45:01,225-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [49723e73] FINISH, GetHardwareInfoVDSCommand, log id: 726c922f
>2018-05-29 15:45:01,239-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [328ecb60] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:45:01,247-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [dc7d39e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:45:01,253-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [49e0f815] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:45:01,264-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [49e0f815] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6467ea21
>2018-05-29 15:45:01,645-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [49e0f815] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:45:01,645-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [49e0f815] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6467ea21
>2018-05-29 15:45:01,649-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [49e0f815] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:45:01,650-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [49e0f815] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:45:01,654-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [49e0f815] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:45:01,654-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [49e0f815] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:45:01,658-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [49e0f815] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:45:01,659-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [49e0f815] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:45:01,662-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [49e0f815] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:45:01,664-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [49e0f815] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 49e0f815, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:45:01,670-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3b7b775
>2018-05-29 15:45:01,671-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:45:01,768-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:45:01,768-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:45:01,768-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:45:01,768-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:45:01,768-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3b7b775
>2018-05-29 15:45:01,768-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:45:01,781-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [6278c01d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:45:01,782-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6278c01d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2619c3a7
>2018-05-29 15:45:01,785-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6278c01d] FINISH, SetVdsStatusVDSCommand, log id: 2619c3a7
>2018-05-29 15:45:01,816-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6278c01d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6278c01d, Job ID: d5cbe33f-ed85-4aad-8889-9e60d8fd65f8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:45:01,821-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6278c01d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: dc7d39e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:45:01,823-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6278c01d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: dc7d39e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:45:01,834-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [52db3ed7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:45:01,835-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [52db3ed7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:45:03,399-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3eb6194f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 37122590
>2018-05-29 15:45:03,401-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3eb6194f] FINISH, GetHardwareInfoVDSCommand, log id: 37122590
>2018-05-29 15:45:03,416-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [26a881e9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:45:03,420-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [2e96624] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:45:03,425-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5d9aa8c8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:45:03,437-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5d9aa8c8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: df606ae
>2018-05-29 15:45:04,815-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d9aa8c8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:45:04,815-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5d9aa8c8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: df606ae
>2018-05-29 15:45:04,819-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d9aa8c8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:45:04,820-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5d9aa8c8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:45:04,824-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d9aa8c8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:45:04,824-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5d9aa8c8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:45:04,828-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d9aa8c8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:45:04,829-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5d9aa8c8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:45:04,832-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5d9aa8c8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:45:04,835-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d9aa8c8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5d9aa8c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:45:04,842-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 26489997
>2018-05-29 15:45:04,842-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:45:04,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:45:04,940-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:45:04,941-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:45:04,941-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:45:04,941-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 26489997
>2018-05-29 15:45:04,941-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:45:04,954-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [3080b85d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:45:04,955-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3080b85d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42691712
>2018-05-29 15:45:04,958-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3080b85d] FINISH, SetVdsStatusVDSCommand, log id: 42691712
>2018-05-29 15:45:04,966-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3080b85d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3080b85d, Job ID: b70f5b50-29ae-430e-96d3-04c6636f26df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:45:04,970-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3080b85d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2e96624, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:45:04,972-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3080b85d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2e96624, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:45:04,983-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5d6802b7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:45:04,984-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5d6802b7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [67c6b3b6] Autorecovering 2 hosts
>2018-05-29 15:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [67c6b3b6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:50:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [28b04aa8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:50:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [28b04aa8] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:50:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [28b04aa8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:50:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [28b04aa8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:50:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [28b04aa8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4e5c8f9b
>2018-05-29 15:50:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [28b04aa8] FINISH, SetVdsStatusVDSCommand, log id: 4e5c8f9b
>2018-05-29 15:50:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [28b04aa8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:50:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [28b04aa8] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:50:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [28b04aa8] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:50:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [53492d19] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [53492d19] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [53492d19] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [53492d19] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:50:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [53492d19] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 114c958f
>2018-05-29 15:50:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [53492d19] FINISH, SetVdsStatusVDSCommand, log id: 114c958f
>2018-05-29 15:50:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [53492d19] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7aafd7c0
>2018-05-29 15:50:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [53492d19] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7aafd7c0
>2018-05-29 15:50:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [53492d19] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:50:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [53492d19] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:50:02,514-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3dde8b2f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2b698320
>2018-05-29 15:50:02,516-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3dde8b2f] FINISH, GetHardwareInfoVDSCommand, log id: 2b698320
>2018-05-29 15:50:02,532-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [1224cb10] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:50:02,540-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [10dbee0b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:50:02,545-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [c30c7ba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:50:02,551-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [c30c7ba] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f7b84a4
>2018-05-29 15:50:02,637-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [33980152] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 69eca898
>2018-05-29 15:50:02,639-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [33980152] FINISH, GetHardwareInfoVDSCommand, log id: 69eca898
>2018-05-29 15:50:02,652-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [74159147] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:50:02,656-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [60a1530d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:50:02,661-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1628c7e6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:50:02,667-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1628c7e6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 24ea45b6
>2018-05-29 15:50:02,923-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [c30c7ba] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:50:02,923-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [c30c7ba] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6f7b84a4
>2018-05-29 15:50:02,927-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [c30c7ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:50:02,928-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [c30c7ba] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:50:02,932-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [c30c7ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:50:02,932-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [c30c7ba] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:50:02,936-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [c30c7ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:50:02,937-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [c30c7ba] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:50:02,940-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [c30c7ba] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:50:02,942-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [c30c7ba] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: c30c7ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:50:02,945-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 67ef1256
>2018-05-29 15:50:02,945-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:50:03,035-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1628c7e6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:50:03,035-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1628c7e6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 24ea45b6
>2018-05-29 15:50:03,039-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1628c7e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:50:03,040-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1628c7e6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:50:03,040-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:50:03,040-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:50:03,040-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:50:03,040-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:50:03,040-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 67ef1256
>2018-05-29 15:50:03,040-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:50:03,044-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1628c7e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:50:03,045-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1628c7e6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:50:03,048-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1628c7e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:50:03,049-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1628c7e6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:50:03,053-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1628c7e6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:50:03,054-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [4a15802a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:50:03,055-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4a15802a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b398117
>2018-05-29 15:50:03,056-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1628c7e6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1628c7e6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:50:03,058-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4a15802a] FINISH, SetVdsStatusVDSCommand, log id: 1b398117
>2018-05-29 15:50:03,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 691ee5f9
>2018-05-29 15:50:03,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:50:03,090-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a15802a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4a15802a, Job ID: 554a89cf-2774-4566-a643-4608acb59057, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:50:03,094-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a15802a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 10dbee0b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:50:03,096-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a15802a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 10dbee0b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:50:03,107-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [33b40f53] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:50:03,108-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [33b40f53] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:50:03,167-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:50:03,167-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:50:03,167-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:50:03,167-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:50:03,167-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 691ee5f9
>2018-05-29 15:50:03,167-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:50:03,180-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [7d7b8a71] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:50:03,182-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7d7b8a71] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ec6f657
>2018-05-29 15:50:03,184-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7d7b8a71] FINISH, SetVdsStatusVDSCommand, log id: 2ec6f657
>2018-05-29 15:50:03,189-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d7b8a71] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7d7b8a71, Job ID: fd608107-724a-481d-97c0-395ce46bd619, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:50:03,194-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d7b8a71] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 60a1530d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:50:03,196-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d7b8a71] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 60a1530d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:50:03,207-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [42c5a076] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:50:03,208-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [42c5a076] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:54:43,111-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler1) [36e40224] Attempting to update VMs/Templates Ovf.
>2018-05-29 15:54:43,113-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 15:54:43,113-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 15:54:43,114-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:54:43,118-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 15:54:43,122-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 15:54:43,122-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 15:54:43,122-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 15:54:43,122-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 15:54:43,123-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 15:54:43,123-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 15:55:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7654db52] Autorecovering 2 hosts
>2018-05-29 15:55:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7654db52] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:55:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c0edefc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:55:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c0edefc] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:55:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c0edefc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:55:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c0edefc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:55:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5c0edefc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4892dc8c
>2018-05-29 15:55:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5c0edefc] FINISH, SetVdsStatusVDSCommand, log id: 4892dc8c
>2018-05-29 15:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c0edefc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:55:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c0edefc] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 15:55:00,070-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5c0edefc] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:55:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2e0a378c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:55:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2e0a378c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 15:55:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2e0a378c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:55:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2e0a378c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:55:00,084-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2e0a378c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 79cc579
>2018-05-29 15:55:00,086-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2e0a378c] FINISH, SetVdsStatusVDSCommand, log id: 79cc579
>2018-05-29 15:55:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [2e0a378c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5a85be57
>2018-05-29 15:55:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [2e0a378c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5a85be57
>2018-05-29 15:55:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2e0a378c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 15:55:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2e0a378c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 15:55:00,774-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7d6d0714] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7609b05d
>2018-05-29 15:55:00,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7d6d0714] FINISH, GetHardwareInfoVDSCommand, log id: 7609b05d
>2018-05-29 15:55:00,789-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [52101437] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:55:00,798-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [46e25abe] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:55:00,804-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [74462c64] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:55:00,810-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [74462c64] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3cf08468
>2018-05-29 15:55:00,823-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3341c7a0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 679c4fe8
>2018-05-29 15:55:00,825-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3341c7a0] FINISH, GetHardwareInfoVDSCommand, log id: 679c4fe8
>2018-05-29 15:55:00,837-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [3c15e46d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:55:00,842-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [6190ab55] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:55:00,847-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [4f4c45f2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 15:55:00,853-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [4f4c45f2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5a73d93
>2018-05-29 15:55:01,186-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [74462c64] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:55:01,186-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [74462c64] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3cf08468
>2018-05-29 15:55:01,190-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [74462c64] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:55:01,191-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [74462c64] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:55:01,195-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [74462c64] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:55:01,196-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [74462c64] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:55:01,199-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [74462c64] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:55:01,200-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [74462c64] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:55:01,203-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [74462c64] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:55:01,205-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [74462c64] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 74462c64, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:55:01,209-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: df31f54
>2018-05-29 15:55:01,209-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:55:01,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4f4c45f2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 15:55:01,224-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [4f4c45f2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5a73d93
>2018-05-29 15:55:01,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4f4c45f2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:55:01,229-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [4f4c45f2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:55:01,233-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4f4c45f2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:55:01,234-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [4f4c45f2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:55:01,237-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4f4c45f2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 15:55:01,238-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [4f4c45f2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 15:55:01,242-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [4f4c45f2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 15:55:01,244-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4f4c45f2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4f4c45f2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 15:55:01,309-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:55:01,309-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:55:01,309-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:55:01,309-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:55:01,309-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: df31f54
>2018-05-29 15:55:01,309-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:55:01,311-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 545af1c9
>2018-05-29 15:55:01,311-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 15:55:01,322-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [131f1ef1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:55:01,324-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [131f1ef1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146bb75f
>2018-05-29 15:55:01,326-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [131f1ef1] FINISH, SetVdsStatusVDSCommand, log id: 146bb75f
>2018-05-29 15:55:01,331-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [131f1ef1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 131f1ef1, Job ID: bf282c6c-0b02-428b-877d-ed51d05182cd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:55:01,336-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [131f1ef1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 46e25abe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:55:01,338-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [131f1ef1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 46e25abe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:55:01,349-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [4244e574] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 15:55:01,350-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [4244e574] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 15:55:01,411-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:55:01,411-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 15:55:01,411-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 15:55:01,411-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 15:55:01,411-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 545af1c9
>2018-05-29 15:55:01,411-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 15:55:01,426-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [44abec7b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:55:01,427-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [44abec7b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: adb7688
>2018-05-29 15:55:01,429-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [44abec7b] FINISH, SetVdsStatusVDSCommand, log id: adb7688
>2018-05-29 15:55:01,435-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [44abec7b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 44abec7b, Job ID: c4ad25b7-5326-4c3a-a2f2-b0e3032423c7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 15:55:01,440-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [44abec7b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6190ab55, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 15:55:01,442-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [44abec7b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6190ab55, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 15:55:01,454-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [5c05533] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 15:55:01,455-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [5c05533] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5f41ebf0] Autorecovering 2 hosts
>2018-05-29 16:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5f41ebf0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:00:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [547d9d05] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [547d9d05] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [547d9d05] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [547d9d05] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:00:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [547d9d05] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1ecf3ef2
>2018-05-29 16:00:02,000-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [547d9d05] FINISH, SetVdsStatusVDSCommand, log id: 1ecf3ef2
>2018-05-29 16:00:02,002-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [547d9d05] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:00:02,004-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [547d9d05] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:00:02,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [547d9d05] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:00:02,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [496ee15c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:00:02,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [496ee15c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:00:02,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [496ee15c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:00:02,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [496ee15c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:00:02,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [496ee15c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5aee52e2
>2018-05-29 16:00:02,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [496ee15c] FINISH, SetVdsStatusVDSCommand, log id: 5aee52e2
>2018-05-29 16:00:02,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [496ee15c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6dbc773f
>2018-05-29 16:00:02,074-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [496ee15c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6dbc773f
>2018-05-29 16:00:02,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [496ee15c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:00:02,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [496ee15c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:00:04,988-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5b1cf189] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 348be048
>2018-05-29 16:00:04,989-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5b1cf189] FINISH, GetHardwareInfoVDSCommand, log id: 348be048
>2018-05-29 16:00:05,004-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [717bada0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:00:05,013-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [52978606] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:00:05,018-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4ab5ceae] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:00:05,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4ab5ceae] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 174d5502
>2018-05-29 16:00:05,250-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [28a6ae81] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: ca381cb
>2018-05-29 16:00:05,252-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [28a6ae81] FINISH, GetHardwareInfoVDSCommand, log id: ca381cb
>2018-05-29 16:00:05,265-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [c02b904] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:00:05,274-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [38164f3d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:00:05,279-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3f004761] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:00:05,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3f004761] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: ba3144d
>2018-05-29 16:00:05,458-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4ab5ceae] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:00:05,458-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4ab5ceae] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 174d5502
>2018-05-29 16:00:05,462-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4ab5ceae] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:00:05,463-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4ab5ceae] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:00:05,467-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4ab5ceae] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:00:05,468-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4ab5ceae] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:00:05,471-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4ab5ceae] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:00:05,472-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4ab5ceae] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:00:05,475-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4ab5ceae] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:00:05,477-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4ab5ceae] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4ab5ceae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:00:05,484-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 52589623
>2018-05-29 16:00:05,484-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:00:05,581-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:00:05,581-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:00:05,581-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:00:05,581-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:00:05,581-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 52589623
>2018-05-29 16:00:05,582-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:00:05,595-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [2b2a2d84] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:00:05,597-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2b2a2d84] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 743f0516
>2018-05-29 16:00:05,599-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2b2a2d84] FINISH, SetVdsStatusVDSCommand, log id: 743f0516
>2018-05-29 16:00:05,631-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2b2a2d84] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2b2a2d84, Job ID: 33c9828d-4973-4b25-9e53-5ed70ea71970, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:00:05,636-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2b2a2d84] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 52978606, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:00:05,638-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2b2a2d84] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 52978606, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:00:05,650-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [9622801] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:00:05,651-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [9622801] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:00:05,654-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3f004761] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:00:05,654-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3f004761] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: ba3144d
>2018-05-29 16:00:05,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3f004761] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:00:05,659-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3f004761] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:00:05,663-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3f004761] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:00:05,664-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3f004761] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:00:05,668-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3f004761] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:00:05,669-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3f004761] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:00:05,672-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3f004761] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:00:05,674-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3f004761] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3f004761, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:00:05,677-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 17859ada
>2018-05-29 16:00:05,677-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:00:05,775-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:00:05,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:00:05,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:00:05,775-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:00:05,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 17859ada
>2018-05-29 16:00:05,775-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:00:05,789-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [66ef70a7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:00:05,790-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [66ef70a7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f4cd28
>2018-05-29 16:00:05,792-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [66ef70a7] FINISH, SetVdsStatusVDSCommand, log id: f4cd28
>2018-05-29 16:00:05,798-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [66ef70a7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 66ef70a7, Job ID: 9dd3db22-6c6a-4706-be66-6173ea6af62e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:00:05,802-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [66ef70a7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 38164f3d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:00:05,805-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [66ef70a7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 38164f3d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:00:05,815-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [6998dd5d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:00:05,816-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [6998dd5d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [4adff942] Autorecovering 2 hosts
>2018-05-29 16:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [4adff942] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:05:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [23ce6d26] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:05:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [23ce6d26] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:05:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [23ce6d26] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:05:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [23ce6d26] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:05:00,050-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [23ce6d26] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 219e3dd7
>2018-05-29 16:05:00,053-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [23ce6d26] FINISH, SetVdsStatusVDSCommand, log id: 219e3dd7
>2018-05-29 16:05:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [23ce6d26] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:05:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [23ce6d26] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:05:00,056-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [23ce6d26] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:05:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [51554fc6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:05:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [51554fc6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:05:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [51554fc6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:05:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [51554fc6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:05:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [51554fc6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2dcf9d64
>2018-05-29 16:05:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [51554fc6] FINISH, SetVdsStatusVDSCommand, log id: 2dcf9d64
>2018-05-29 16:05:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [51554fc6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 673465b5
>2018-05-29 16:05:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [51554fc6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 673465b5
>2018-05-29 16:05:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [51554fc6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:05:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [51554fc6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:05:00,339-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [14fd3aee] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 577fc57d
>2018-05-29 16:05:00,341-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [14fd3aee] FINISH, GetHardwareInfoVDSCommand, log id: 577fc57d
>2018-05-29 16:05:00,353-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3b8362f5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:05:00,361-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [7e052f85] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:05:00,366-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5160606a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:05:00,373-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5160606a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 20634e83
>2018-05-29 16:05:00,425-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [75092fd1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6583c1ce
>2018-05-29 16:05:00,427-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [75092fd1] FINISH, GetHardwareInfoVDSCommand, log id: 6583c1ce
>2018-05-29 16:05:00,439-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [e81b9b4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:05:00,443-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3188594e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:05:00,448-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [e6942dc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:05:00,453-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [e6942dc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22bcc966
>2018-05-29 16:05:00,762-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5160606a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:05:00,762-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5160606a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 20634e83
>2018-05-29 16:05:00,766-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5160606a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:05:00,767-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5160606a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:05:00,771-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5160606a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:05:00,771-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5160606a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:05:00,775-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5160606a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:05:00,776-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5160606a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:05:00,779-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5160606a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:05:00,782-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5160606a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5160606a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:05:00,785-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 183757a2
>2018-05-29 16:05:00,785-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:05:00,830-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e6942dc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:05:00,830-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [e6942dc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 22bcc966
>2018-05-29 16:05:00,833-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e6942dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:05:00,834-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [e6942dc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:05:00,838-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e6942dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:05:00,839-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [e6942dc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:05:00,843-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e6942dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:05:00,844-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [e6942dc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:05:00,846-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [e6942dc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:05:00,849-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e6942dc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: e6942dc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:05:00,884-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:05:00,884-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:05:00,884-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:05:00,884-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:05:00,884-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 183757a2
>2018-05-29 16:05:00,884-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:05:00,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 215f29f
>2018-05-29 16:05:00,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:05:00,897-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [608804f6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:05:00,898-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [608804f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2977c589
>2018-05-29 16:05:00,901-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [608804f6] FINISH, SetVdsStatusVDSCommand, log id: 2977c589
>2018-05-29 16:05:00,930-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [608804f6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 608804f6, Job ID: 57d26a64-4d8a-437e-a69a-9a2489154ac8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:05:00,935-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [608804f6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7e052f85, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:05:00,937-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [608804f6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7e052f85, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:05:00,947-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6233025a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:05:00,948-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6233025a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:05:00,986-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:05:00,986-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:05:00,986-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:05:00,986-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:05:00,987-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 215f29f
>2018-05-29 16:05:00,987-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:05:00,999-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [3aa50b1c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:05:01,000-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3aa50b1c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72e4bcac
>2018-05-29 16:05:01,002-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3aa50b1c] FINISH, SetVdsStatusVDSCommand, log id: 72e4bcac
>2018-05-29 16:05:01,008-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3aa50b1c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3aa50b1c, Job ID: 413fe6ee-5596-4a10-9c32-cff3be3b249f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:05:01,012-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3aa50b1c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3188594e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:05:01,014-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3aa50b1c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3188594e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:05:01,024-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [47dfa90f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:05:01,025-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [47dfa90f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [39a55cdc] Autorecovering 2 hosts
>2018-05-29 16:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [39a55cdc] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:10:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4eb008a3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:10:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4eb008a3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:10:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4eb008a3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:10:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4eb008a3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:10:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4eb008a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 605b523f
>2018-05-29 16:10:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4eb008a3] FINISH, SetVdsStatusVDSCommand, log id: 605b523f
>2018-05-29 16:10:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4eb008a3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:10:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4eb008a3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:10:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [4eb008a3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:10:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3fd2fd00] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:10:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3fd2fd00] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:10:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3fd2fd00] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:10:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3fd2fd00] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:10:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3fd2fd00] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4f65207b
>2018-05-29 16:10:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3fd2fd00] FINISH, SetVdsStatusVDSCommand, log id: 4f65207b
>2018-05-29 16:10:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [3fd2fd00] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3f43b440
>2018-05-29 16:10:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [3fd2fd00] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3f43b440
>2018-05-29 16:10:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3fd2fd00] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:10:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3fd2fd00] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:10:01,543-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [47bc3ac7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6d262e76
>2018-05-29 16:10:01,545-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [47bc3ac7] FINISH, GetHardwareInfoVDSCommand, log id: 6d262e76
>2018-05-29 16:10:01,557-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [46487940] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:10:01,566-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [f1614e7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:10:01,571-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [680e8482] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:10:01,582-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [680e8482] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3cd0839
>2018-05-29 16:10:01,688-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [5401d6ad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 77d8aefe
>2018-05-29 16:10:01,690-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [5401d6ad] FINISH, GetHardwareInfoVDSCommand, log id: 77d8aefe
>2018-05-29 16:10:01,703-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [1e52f455] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:10:01,707-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [7a4a70f6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:10:01,712-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [74211327] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:10:01,719-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [74211327] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 43d0fb39
>2018-05-29 16:10:01,961-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [680e8482] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:10:01,961-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [680e8482] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3cd0839
>2018-05-29 16:10:01,965-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [680e8482] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:10:01,966-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [680e8482] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:10:01,970-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [680e8482] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:10:01,971-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [680e8482] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:10:01,975-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [680e8482] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:10:01,976-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [680e8482] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:10:01,978-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [680e8482] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:10:01,981-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [680e8482] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 680e8482, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:10:01,988-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1586e3e0
>2018-05-29 16:10:01,988-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:10:02,084-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [74211327] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:10:02,084-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [74211327] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 43d0fb39
>2018-05-29 16:10:02,086-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:10:02,086-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:10:02,086-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:10:02,086-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:10:02,086-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1586e3e0
>2018-05-29 16:10:02,086-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:10:02,088-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [74211327] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:10:02,089-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [74211327] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:10:02,092-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [74211327] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:10:02,093-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [74211327] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:10:02,098-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [74211327] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:10:02,098-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [74211327] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:10:02,099-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [34dd554a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:10:02,101-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [34dd554a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 122acba5
>2018-05-29 16:10:02,102-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [74211327] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:10:02,103-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [34dd554a] FINISH, SetVdsStatusVDSCommand, log id: 122acba5
>2018-05-29 16:10:02,104-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [74211327] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 74211327, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:10:02,111-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34dd554a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 34dd554a, Job ID: 2282faba-cf02-4841-8e33-63609d981d2d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:10:02,112-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 53bc0fd1
>2018-05-29 16:10:02,112-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:10:02,116-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34dd554a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: f1614e7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:10:02,118-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34dd554a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: f1614e7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:10:02,130-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [29183894] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:10:02,131-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [29183894] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:10:02,208-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:10:02,208-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:10:02,208-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:10:02,208-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:10:02,208-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 53bc0fd1
>2018-05-29 16:10:02,208-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:10:02,221-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [25fa59bb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:10:02,223-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [25fa59bb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a9a6052
>2018-05-29 16:10:02,225-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [25fa59bb] FINISH, SetVdsStatusVDSCommand, log id: 5a9a6052
>2018-05-29 16:10:02,255-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [25fa59bb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 25fa59bb, Job ID: d5dbdd28-c2e0-44dc-a853-7424bbf3f278, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:10:02,259-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [25fa59bb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7a4a70f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:10:02,261-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [25fa59bb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7a4a70f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:10:02,272-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [548d2db9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:10:02,273-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [548d2db9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [491e366] Autorecovering 2 hosts
>2018-05-29 16:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [491e366] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:15:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [17e8824f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:15:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [17e8824f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:15:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [17e8824f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:15:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [17e8824f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:15:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [17e8824f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c10554e
>2018-05-29 16:15:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [17e8824f] FINISH, SetVdsStatusVDSCommand, log id: 2c10554e
>2018-05-29 16:15:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [17e8824f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:15:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [17e8824f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:15:00,082-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [17e8824f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:15:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2731b2de] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:15:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2731b2de] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:15:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2731b2de] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:15:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2731b2de] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:15:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2731b2de] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1e6757b3
>2018-05-29 16:15:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2731b2de] FINISH, SetVdsStatusVDSCommand, log id: 1e6757b3
>2018-05-29 16:15:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [2731b2de] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 71ab610b
>2018-05-29 16:15:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [2731b2de] FINISH, SetHaMaintenanceModeVDSCommand, log id: 71ab610b
>2018-05-29 16:15:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2731b2de] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:15:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2731b2de] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:15:02,816-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [33b0d5d9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 60c3c86
>2018-05-29 16:15:02,817-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [33b0d5d9] FINISH, GetHardwareInfoVDSCommand, log id: 60c3c86
>2018-05-29 16:15:02,832-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [733227b0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:15:02,840-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [35388ecc] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:15:02,846-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [671f9a16] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:15:02,858-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [671f9a16] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 103bbca7
>2018-05-29 16:15:02,930-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7e4443bf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 49788350
>2018-05-29 16:15:02,932-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7e4443bf] FINISH, GetHardwareInfoVDSCommand, log id: 49788350
>2018-05-29 16:15:02,944-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [60d7c783] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:15:02,949-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [565a242] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:15:02,954-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4d80c05c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:15:02,960-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4d80c05c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 15b9b2c5
>2018-05-29 16:15:03,240-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [671f9a16] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:15:03,240-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [671f9a16] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 103bbca7
>2018-05-29 16:15:03,244-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [671f9a16] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:15:03,245-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [671f9a16] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:15:03,248-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [671f9a16] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:15:03,249-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [671f9a16] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:15:03,253-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [671f9a16] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:15:03,254-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [671f9a16] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:15:03,256-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [671f9a16] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:15:03,259-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [671f9a16] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 671f9a16, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:15:03,265-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 19cb8f4b
>2018-05-29 16:15:03,265-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:15:03,335-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4d80c05c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:15:03,335-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4d80c05c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 15b9b2c5
>2018-05-29 16:15:03,339-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4d80c05c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:15:03,340-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4d80c05c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:15:03,344-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4d80c05c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:15:03,344-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4d80c05c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:15:03,348-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4d80c05c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:15:03,349-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4d80c05c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:15:03,352-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4d80c05c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:15:03,354-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4d80c05c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d80c05c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:15:03,363-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:15:03,363-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:15:03,363-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:15:03,363-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:15:03,363-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 19cb8f4b
>2018-05-29 16:15:03,363-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:15:03,365-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 48386d8
>2018-05-29 16:15:03,365-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:15:03,377-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [4fa0575c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:15:03,378-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4fa0575c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4a511eb4
>2018-05-29 16:15:03,380-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4fa0575c] FINISH, SetVdsStatusVDSCommand, log id: 4a511eb4
>2018-05-29 16:15:03,411-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4fa0575c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4fa0575c, Job ID: 74e39d50-21b9-434b-b50a-c51635d488b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:15:03,416-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4fa0575c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 35388ecc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:15:03,419-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4fa0575c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 35388ecc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:15:03,430-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [73b073e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:15:03,431-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [73b073e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:15:03,468-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:15:03,468-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:15:03,468-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:15:03,468-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:15:03,468-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 48386d8
>2018-05-29 16:15:03,468-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:15:03,482-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [124f47a0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:15:03,483-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [124f47a0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46dab786
>2018-05-29 16:15:03,485-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [124f47a0] FINISH, SetVdsStatusVDSCommand, log id: 46dab786
>2018-05-29 16:15:03,491-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [124f47a0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 124f47a0, Job ID: e4d72331-8c7a-45f8-89af-e27566a71c55, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:15:03,495-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [124f47a0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 565a242, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:15:03,498-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [124f47a0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 565a242, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:15:03,508-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [66169357] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:15:03,509-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [66169357] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:20:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [209b15a0] Autorecovering 2 hosts
>2018-05-29 16:20:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [209b15a0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:20:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4ca8d7a2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:20:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4ca8d7a2] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:20:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4ca8d7a2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:20:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4ca8d7a2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:20:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4ca8d7a2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 98ce42b
>2018-05-29 16:20:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4ca8d7a2] FINISH, SetVdsStatusVDSCommand, log id: 98ce42b
>2018-05-29 16:20:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4ca8d7a2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:20:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4ca8d7a2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:20:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [4ca8d7a2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:20:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4a008bde] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:20:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4a008bde] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:20:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4a008bde] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:20:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4a008bde] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:20:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4a008bde] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 67c8688e
>2018-05-29 16:20:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4a008bde] FINISH, SetVdsStatusVDSCommand, log id: 67c8688e
>2018-05-29 16:20:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [4a008bde] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 574649b0
>2018-05-29 16:20:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [4a008bde] FINISH, SetHaMaintenanceModeVDSCommand, log id: 574649b0
>2018-05-29 16:20:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4a008bde] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:20:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4a008bde] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:20:00,992-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [14079fd8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 74448439
>2018-05-29 16:20:00,994-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [14079fd8] FINISH, GetHardwareInfoVDSCommand, log id: 74448439
>2018-05-29 16:20:01,007-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7041252f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:20:01,012-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [237ef9e2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:20:01,017-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [54ae9f62] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:20:01,023-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [54ae9f62] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4638a16d
>2018-05-29 16:20:01,164-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [32688010] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 35b7f834
>2018-05-29 16:20:01,166-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [32688010] FINISH, GetHardwareInfoVDSCommand, log id: 35b7f834
>2018-05-29 16:20:01,178-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [5e9d3da2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:20:01,182-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [6635f5ac] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:20:01,187-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [376fe387] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:20:01,193-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [376fe387] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 297e6c7d
>2018-05-29 16:20:01,398-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [54ae9f62] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:20:01,398-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [54ae9f62] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4638a16d
>2018-05-29 16:20:01,402-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [54ae9f62] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:20:01,403-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [54ae9f62] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:20:01,406-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [54ae9f62] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:20:01,407-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [54ae9f62] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:20:01,411-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [54ae9f62] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:20:01,412-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [54ae9f62] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:20:01,415-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [54ae9f62] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:20:01,418-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [54ae9f62] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 54ae9f62, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:20:01,421-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2581ba0c
>2018-05-29 16:20:01,421-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:20:01,515-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:20:01,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:20:01,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:20:01,515-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:20:01,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2581ba0c
>2018-05-29 16:20:01,515-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:20:01,528-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [7c7e12f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:20:01,530-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7c7e12f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f5e077e
>2018-05-29 16:20:01,532-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7c7e12f] FINISH, SetVdsStatusVDSCommand, log id: f5e077e
>2018-05-29 16:20:01,560-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7c7e12f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7c7e12f, Job ID: b174ac97-88cc-4b8f-9459-6c3ccc71c219, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:20:01,565-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7c7e12f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 237ef9e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:20:01,567-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7c7e12f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 237ef9e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:20:01,578-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [8148f28] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:20:01,579-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [8148f28] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:20:02,558-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [376fe387] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:20:02,558-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [376fe387] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 297e6c7d
>2018-05-29 16:20:02,562-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [376fe387] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:20:02,563-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [376fe387] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:20:02,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [376fe387] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:20:02,567-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [376fe387] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:20:02,571-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [376fe387] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:20:02,572-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [376fe387] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:20:02,575-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [376fe387] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:20:02,577-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [376fe387] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 376fe387, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:20:02,581-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 671bd7d0
>2018-05-29 16:20:02,581-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:20:02,677-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:20:02,677-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:20:02,677-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:20:02,677-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:20:02,677-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 671bd7d0
>2018-05-29 16:20:02,677-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:20:02,690-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2891b3f1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:20:02,691-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2891b3f1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 200b7280
>2018-05-29 16:20:02,693-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2891b3f1] FINISH, SetVdsStatusVDSCommand, log id: 200b7280
>2018-05-29 16:20:02,699-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2891b3f1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2891b3f1, Job ID: a1711388-a525-4b76-9651-2ab50ef90ea1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:20:02,703-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2891b3f1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6635f5ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:20:02,706-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2891b3f1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6635f5ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:20:02,716-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [34f2f5ed] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:20:02,717-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [34f2f5ed] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [144179a1] Autorecovering 2 hosts
>2018-05-29 16:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [144179a1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:25:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3d4a2aa5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:25:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3d4a2aa5] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:25:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3d4a2aa5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:25:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3d4a2aa5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:25:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3d4a2aa5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5cb4cdb9
>2018-05-29 16:25:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3d4a2aa5] FINISH, SetVdsStatusVDSCommand, log id: 5cb4cdb9
>2018-05-29 16:25:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3d4a2aa5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:25:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3d4a2aa5] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:25:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3d4a2aa5] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:25:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [633aaf78] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:25:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [633aaf78] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:25:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [633aaf78] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:25:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [633aaf78] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:25:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [633aaf78] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3fe5fe88
>2018-05-29 16:25:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [633aaf78] FINISH, SetVdsStatusVDSCommand, log id: 3fe5fe88
>2018-05-29 16:25:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [633aaf78] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4156c074
>2018-05-29 16:25:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [633aaf78] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4156c074
>2018-05-29 16:25:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [633aaf78] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:25:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [633aaf78] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:25:00,484-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [7d3de395] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 369157e0
>2018-05-29 16:25:00,486-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [7d3de395] FINISH, GetHardwareInfoVDSCommand, log id: 369157e0
>2018-05-29 16:25:00,499-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [6ffa23e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:25:00,507-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [7a393305] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:25:00,512-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4d1e54d3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:25:00,569-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4d1e54d3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 31807b94
>2018-05-29 16:25:00,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4d1e54d3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:25:00,944-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4d1e54d3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 31807b94
>2018-05-29 16:25:00,948-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4d1e54d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:25:00,949-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4d1e54d3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:25:00,952-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4d1e54d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:25:00,953-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4d1e54d3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:25:00,957-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4d1e54d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:25:00,957-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4d1e54d3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:25:00,960-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4d1e54d3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:25:00,963-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4d1e54d3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d1e54d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:25:00,969-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 200e0ebb
>2018-05-29 16:25:00,969-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:25:01,069-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:25:01,069-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:25:01,069-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:25:01,069-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:25:01,069-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 200e0ebb
>2018-05-29 16:25:01,069-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:25:01,083-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [3fc66d0b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:25:01,084-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3fc66d0b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 633443cd
>2018-05-29 16:25:01,086-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3fc66d0b] FINISH, SetVdsStatusVDSCommand, log id: 633443cd
>2018-05-29 16:25:01,118-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3fc66d0b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3fc66d0b, Job ID: 2cbdbfcf-68dd-48d4-a8c7-1970f5ed7e78, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:25:01,123-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3fc66d0b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7a393305, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:25:01,125-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3fc66d0b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7a393305, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:25:01,136-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [32a25855] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:25:01,137-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [32a25855] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:25:02,188-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [719827d4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 8a0a309
>2018-05-29 16:25:02,190-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [719827d4] FINISH, GetHardwareInfoVDSCommand, log id: 8a0a309
>2018-05-29 16:25:02,203-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [162508b1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:25:02,207-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [62cd7dc7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:25:02,212-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3aaaf4fd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:25:02,219-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3aaaf4fd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f7af0b5
>2018-05-29 16:25:02,601-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3aaaf4fd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:25:02,601-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3aaaf4fd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f7af0b5
>2018-05-29 16:25:02,605-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3aaaf4fd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:25:02,606-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3aaaf4fd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:25:02,610-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3aaaf4fd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:25:02,610-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3aaaf4fd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:25:02,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3aaaf4fd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:25:02,615-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3aaaf4fd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:25:02,617-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3aaaf4fd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:25:02,620-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3aaaf4fd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3aaaf4fd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:25:02,623-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 56ca181c
>2018-05-29 16:25:02,623-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:25:02,721-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:25:02,721-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:25:02,722-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:25:02,722-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:25:02,722-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 56ca181c
>2018-05-29 16:25:02,722-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:25:02,735-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4f0c0676] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:25:02,736-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4f0c0676] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61f3ae3d
>2018-05-29 16:25:02,738-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4f0c0676] FINISH, SetVdsStatusVDSCommand, log id: 61f3ae3d
>2018-05-29 16:25:02,744-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f0c0676] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4f0c0676, Job ID: 6dbeac57-789a-4022-ae9e-40440c904d56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:25:02,749-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f0c0676] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 62cd7dc7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:25:02,751-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f0c0676] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 62cd7dc7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:25:02,762-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [34ccda9f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:25:02,763-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [34ccda9f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3ad168e] Autorecovering 2 hosts
>2018-05-29 16:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3ad168e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:30:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7dfbb65e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:30:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7dfbb65e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:30:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7dfbb65e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:30:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7dfbb65e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:30:00,051-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7dfbb65e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2f31bf6b
>2018-05-29 16:30:00,054-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7dfbb65e] FINISH, SetVdsStatusVDSCommand, log id: 2f31bf6b
>2018-05-29 16:30:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7dfbb65e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:30:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7dfbb65e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:30:00,057-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7dfbb65e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:30:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [19ebda78] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:30:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [19ebda78] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:30:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [19ebda78] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:30:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [19ebda78] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:30:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [19ebda78] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d4b6066
>2018-05-29 16:30:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [19ebda78] FINISH, SetVdsStatusVDSCommand, log id: 3d4b6066
>2018-05-29 16:30:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [19ebda78] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 60384b5a
>2018-05-29 16:30:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [19ebda78] FINISH, SetHaMaintenanceModeVDSCommand, log id: 60384b5a
>2018-05-29 16:30:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [19ebda78] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:30:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [19ebda78] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:30:00,381-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [589ef885] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 69efa4e6
>2018-05-29 16:30:00,383-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [589ef885] FINISH, GetHardwareInfoVDSCommand, log id: 69efa4e6
>2018-05-29 16:30:00,396-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [15a8a3f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:30:00,404-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [6979b4cb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:30:00,409-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [c75920] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:30:00,416-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [c75920] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7a1ae9e9
>2018-05-29 16:30:00,795-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c75920] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:30:00,795-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [c75920] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7a1ae9e9
>2018-05-29 16:30:00,799-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c75920] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:30:00,800-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [c75920] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:30:00,804-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c75920] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:30:00,805-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [c75920] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:30:00,809-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c75920] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:30:00,809-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [c75920] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:30:00,812-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [c75920] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:30:00,815-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c75920] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: c75920, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:30:00,817-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 679b7dda
>2018-05-29 16:30:00,818-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:30:00,916-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:30:00,916-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:30:00,916-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:30:00,916-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:30:00,916-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 679b7dda
>2018-05-29 16:30:00,916-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:30:00,929-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [7ca078b0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:30:00,930-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7ca078b0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 65cf1cbf
>2018-05-29 16:30:00,932-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7ca078b0] FINISH, SetVdsStatusVDSCommand, log id: 65cf1cbf
>2018-05-29 16:30:00,961-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ca078b0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7ca078b0, Job ID: f05fef4f-d2ed-47eb-91ec-270d3c06349a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:30:00,966-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ca078b0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6979b4cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:30:00,968-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ca078b0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6979b4cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:30:00,979-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [35c1897f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:30:00,980-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [35c1897f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:30:01,906-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [101168b4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 288ab8c8
>2018-05-29 16:30:01,908-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [101168b4] FINISH, GetHardwareInfoVDSCommand, log id: 288ab8c8
>2018-05-29 16:30:01,922-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [248946c0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:30:01,926-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [4df473b7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:30:01,932-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [365ea687] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:30:01,938-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [365ea687] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5e1fcc09
>2018-05-29 16:30:03,318-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [365ea687] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:30:03,318-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [365ea687] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5e1fcc09
>2018-05-29 16:30:03,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [365ea687] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:30:03,323-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [365ea687] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:30:03,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [365ea687] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:30:03,328-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [365ea687] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:30:03,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [365ea687] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:30:03,333-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [365ea687] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:30:03,336-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [365ea687] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:30:03,338-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [365ea687] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 365ea687, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:30:03,345-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 29caa681
>2018-05-29 16:30:03,345-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:30:03,442-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:30:03,442-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:30:03,442-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:30:03,442-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:30:03,442-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 29caa681
>2018-05-29 16:30:03,442-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:30:03,455-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [6b5fa6a8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:30:03,457-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6b5fa6a8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5101be0f
>2018-05-29 16:30:03,459-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6b5fa6a8] FINISH, SetVdsStatusVDSCommand, log id: 5101be0f
>2018-05-29 16:30:03,491-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6b5fa6a8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6b5fa6a8, Job ID: c85f99f0-5866-4cfe-bdde-e7c4b2e212df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:30:03,495-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6b5fa6a8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4df473b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:30:03,497-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6b5fa6a8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4df473b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:30:03,508-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [46c45a27] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:30:03,510-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [46c45a27] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:35:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3588bf55] Autorecovering 2 hosts
>2018-05-29 16:35:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3588bf55] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:35:00,033-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f2a329] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:35:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f2a329] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:35:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f2a329] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:35:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f2a329] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:35:00,041-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3f2a329] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1d8b7a59
>2018-05-29 16:35:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3f2a329] FINISH, SetVdsStatusVDSCommand, log id: 1d8b7a59
>2018-05-29 16:35:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f2a329] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:35:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f2a329] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:35:00,047-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3f2a329] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:35:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2c2c1d5f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:35:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2c2c1d5f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:35:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2c2c1d5f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:35:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2c2c1d5f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:35:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2c2c1d5f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5d55fc6
>2018-05-29 16:35:00,062-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2c2c1d5f] FINISH, SetVdsStatusVDSCommand, log id: 5d55fc6
>2018-05-29 16:35:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [2c2c1d5f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7aecab0f
>2018-05-29 16:35:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [2c2c1d5f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7aecab0f
>2018-05-29 16:35:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2c2c1d5f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:35:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2c2c1d5f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:35:01,165-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1ed9f88c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6ab486cf
>2018-05-29 16:35:01,166-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1ed9f88c] FINISH, GetHardwareInfoVDSCommand, log id: 6ab486cf
>2018-05-29 16:35:01,179-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [7523ecbf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:35:01,184-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [66b06f36] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:35:01,189-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4d8b4e8f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:35:01,201-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4d8b4e8f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7ce3133e
>2018-05-29 16:35:01,576-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d8b4e8f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:35:01,576-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4d8b4e8f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7ce3133e
>2018-05-29 16:35:01,580-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d8b4e8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:35:01,581-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d8b4e8f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:35:01,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d8b4e8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:35:01,587-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d8b4e8f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:35:01,588-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6a7878cd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3c8b6725
>2018-05-29 16:35:01,590-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6a7878cd] FINISH, GetHardwareInfoVDSCommand, log id: 3c8b6725
>2018-05-29 16:35:01,591-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d8b4e8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:35:01,592-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d8b4e8f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:35:01,596-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4d8b4e8f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:35:01,599-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d8b4e8f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d8b4e8f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:35:01,604-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [497765c5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:35:01,606-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6c7d4304
>2018-05-29 16:35:01,606-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:35:01,613-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [13f626a2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:35:01,619-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [720d011f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:35:01,630-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [720d011f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2ec531a0
>2018-05-29 16:35:01,709-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:35:01,709-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:35:01,709-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:35:01,709-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:35:01,709-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6c7d4304
>2018-05-29 16:35:01,709-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:35:01,722-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [3e5d01cd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:35:01,723-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3e5d01cd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 57ef08ae
>2018-05-29 16:35:01,725-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3e5d01cd] FINISH, SetVdsStatusVDSCommand, log id: 57ef08ae
>2018-05-29 16:35:01,734-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3e5d01cd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3e5d01cd, Job ID: 4c3d3532-fd42-46a4-b5e2-96a4b0ab7576, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:35:01,739-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3e5d01cd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 66b06f36, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:35:01,741-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3e5d01cd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 66b06f36, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:35:01,751-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [86a3c2a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:35:01,752-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [86a3c2a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:35:02,008-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [720d011f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:35:02,008-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [720d011f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2ec531a0
>2018-05-29 16:35:02,012-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [720d011f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:35:02,013-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [720d011f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:35:02,017-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [720d011f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:35:02,018-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [720d011f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:35:02,021-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [720d011f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:35:02,022-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [720d011f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:35:02,025-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [720d011f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:35:02,027-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [720d011f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 720d011f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:35:02,030-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 673a6c4c
>2018-05-29 16:35:02,030-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:35:02,123-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:35:02,123-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:35:02,123-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:35:02,123-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:35:02,123-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 673a6c4c
>2018-05-29 16:35:02,123-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:35:02,136-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [368dfbd4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:35:02,137-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [368dfbd4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4228af8c
>2018-05-29 16:35:02,139-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [368dfbd4] FINISH, SetVdsStatusVDSCommand, log id: 4228af8c
>2018-05-29 16:35:02,144-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [368dfbd4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 368dfbd4, Job ID: ba888fd1-82a8-408c-beaa-ede2714572c9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:35:02,149-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [368dfbd4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 13f626a2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:35:02,151-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [368dfbd4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 13f626a2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:35:02,161-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [2d1b579] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:35:02,162-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [2d1b579] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:40:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1d313705] Autorecovering 2 hosts
>2018-05-29 16:40:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1d313705] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:40:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4790f22a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:40:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4790f22a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:40:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4790f22a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:40:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4790f22a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:40:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4790f22a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6a10ea8c
>2018-05-29 16:40:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4790f22a] FINISH, SetVdsStatusVDSCommand, log id: 6a10ea8c
>2018-05-29 16:40:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4790f22a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:40:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4790f22a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:40:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [4790f22a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:40:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [798ab0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:40:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [798ab0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:40:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [798ab0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:40:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [798ab0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:40:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [798ab0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a7eacee
>2018-05-29 16:40:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [798ab0] FINISH, SetVdsStatusVDSCommand, log id: a7eacee
>2018-05-29 16:40:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [798ab0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 293c2234
>2018-05-29 16:40:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [798ab0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 293c2234
>2018-05-29 16:40:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [798ab0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:40:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [798ab0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:40:02,717-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4ed88466] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6f2283f4
>2018-05-29 16:40:02,719-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4ed88466] FINISH, GetHardwareInfoVDSCommand, log id: 6f2283f4
>2018-05-29 16:40:02,733-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3f85e9e3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:40:02,741-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [3b202975] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:40:02,746-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [74268304] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:40:02,758-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [74268304] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: ec9c7d5
>2018-05-29 16:40:02,820-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [5019642c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 694636f1
>2018-05-29 16:40:02,821-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [5019642c] FINISH, GetHardwareInfoVDSCommand, log id: 694636f1
>2018-05-29 16:40:02,834-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [55067287] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:40:02,838-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [1d3cde0c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:40:02,843-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [dfc4b96] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:40:02,849-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [dfc4b96] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 72e7a2b6
>2018-05-29 16:40:03,133-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [74268304] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:40:03,133-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [74268304] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: ec9c7d5
>2018-05-29 16:40:03,137-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [74268304] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:40:03,138-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [74268304] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:40:03,142-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [74268304] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:40:03,142-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [74268304] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:40:03,146-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [74268304] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:40:03,147-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [74268304] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:40:03,150-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [74268304] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:40:03,153-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [74268304] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 74268304, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:40:03,159-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6e4a4322
>2018-05-29 16:40:03,159-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:40:03,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [dfc4b96] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:40:03,226-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [dfc4b96] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 72e7a2b6
>2018-05-29 16:40:03,229-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [dfc4b96] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:40:03,230-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [dfc4b96] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:40:03,234-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [dfc4b96] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:40:03,235-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [dfc4b96] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:40:03,238-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [dfc4b96] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:40:03,239-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [dfc4b96] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:40:03,242-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [dfc4b96] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:40:03,244-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [dfc4b96] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: dfc4b96, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:40:03,256-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:40:03,256-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:40:03,256-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:40:03,256-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:40:03,256-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6e4a4322
>2018-05-29 16:40:03,256-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:40:03,257-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 179273ab
>2018-05-29 16:40:03,258-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:40:03,269-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [42261683] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:40:03,270-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [42261683] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 777b9c42
>2018-05-29 16:40:03,272-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [42261683] FINISH, SetVdsStatusVDSCommand, log id: 777b9c42
>2018-05-29 16:40:03,304-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [42261683] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 42261683, Job ID: ae4db035-847c-40fe-ae09-72763b7452ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:40:03,308-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [42261683] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b202975, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:40:03,310-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [42261683] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b202975, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:40:03,321-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [3c6ce366] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:40:03,322-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [3c6ce366] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:40:03,360-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-33) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:40:03,360-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:40:03,360-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:40:03,360-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:40:03,360-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] FINISH, ConnectStoragePoolVDSCommand, log id: 179273ab
>2018-05-29 16:40:03,360-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-33) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:40:03,373-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [1902e309] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:40:03,374-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1902e309] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 78f4c3bd
>2018-05-29 16:40:03,376-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1902e309] FINISH, SetVdsStatusVDSCommand, log id: 78f4c3bd
>2018-05-29 16:40:03,382-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1902e309] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1902e309, Job ID: 17277738-04c8-4421-899c-b4a19660aded, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:40:03,386-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1902e309] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d3cde0c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:40:03,388-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1902e309] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d3cde0c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:40:03,398-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [1682215a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:40:03,399-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [1682215a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3300d87b] Autorecovering 2 hosts
>2018-05-29 16:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3300d87b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:45:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [237aeaa1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [237aeaa1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [237aeaa1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [237aeaa1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:45:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [237aeaa1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2bde7ed8
>2018-05-29 16:45:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [237aeaa1] FINISH, SetVdsStatusVDSCommand, log id: 2bde7ed8
>2018-05-29 16:45:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [237aeaa1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:45:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [237aeaa1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:45:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [237aeaa1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:45:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [743c44d3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [743c44d3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [743c44d3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [743c44d3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:45:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [743c44d3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1a0df1be
>2018-05-29 16:45:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [743c44d3] FINISH, SetVdsStatusVDSCommand, log id: 1a0df1be
>2018-05-29 16:45:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [743c44d3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: b172d66
>2018-05-29 16:45:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [743c44d3] FINISH, SetHaMaintenanceModeVDSCommand, log id: b172d66
>2018-05-29 16:45:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [743c44d3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:45:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [743c44d3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:45:00,986-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6ed26e96] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5bab4d4
>2018-05-29 16:45:00,988-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6ed26e96] FINISH, GetHardwareInfoVDSCommand, log id: 5bab4d4
>2018-05-29 16:45:01,001-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [1affc51d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:45:01,010-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [70e30a63] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:45:01,012-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6bd9e4dd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 44900058
>2018-05-29 16:45:01,013-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6bd9e4dd] FINISH, GetHardwareInfoVDSCommand, log id: 44900058
>2018-05-29 16:45:01,016-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7853cff6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:45:01,024-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7853cff6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6fb982f1
>2018-05-29 16:45:01,028-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [2db36ee5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:45:01,036-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [74748c0c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:45:01,042-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3c5e6361] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:45:01,048-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3c5e6361] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 68aa231e
>2018-05-29 16:45:01,396-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7853cff6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:45:01,396-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7853cff6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6fb982f1
>2018-05-29 16:45:01,400-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7853cff6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:45:01,401-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7853cff6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:45:01,405-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7853cff6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:45:01,405-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7853cff6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:45:01,409-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7853cff6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:45:01,410-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7853cff6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:45:01,413-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7853cff6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:45:01,416-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7853cff6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7853cff6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:45:01,422-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4526d006
>2018-05-29 16:45:01,423-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:45:01,431-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c5e6361] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:45:01,431-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3c5e6361] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 68aa231e
>2018-05-29 16:45:01,435-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c5e6361] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:45:01,436-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3c5e6361] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:45:01,439-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c5e6361] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:45:01,440-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3c5e6361] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:45:01,444-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c5e6361] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:45:01,444-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3c5e6361] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:45:01,447-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3c5e6361] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:45:01,450-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c5e6361] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3c5e6361, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:45:01,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:45:01,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:45:01,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:45:01,522-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:45:01,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4526d006
>2018-05-29 16:45:01,522-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:45:01,523-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 64345ac6
>2018-05-29 16:45:01,523-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:45:01,535-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [4a16a74] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:45:01,536-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4a16a74] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4c85abb0
>2018-05-29 16:45:01,539-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4a16a74] FINISH, SetVdsStatusVDSCommand, log id: 4c85abb0
>2018-05-29 16:45:01,570-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4a16a74] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4a16a74, Job ID: d5af2478-562e-48bf-b5e9-f588775c0ce2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:45:01,574-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4a16a74] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 70e30a63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:45:01,576-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4a16a74] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 70e30a63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:45:01,587-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5e4597ec] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:45:01,588-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5e4597ec] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:45:01,623-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:45:01,623-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:45:01,623-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:45:01,623-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:45:01,623-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 64345ac6
>2018-05-29 16:45:01,623-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:45:01,636-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [42928c56] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:45:01,637-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [42928c56] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7522148d
>2018-05-29 16:45:01,639-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [42928c56] FINISH, SetVdsStatusVDSCommand, log id: 7522148d
>2018-05-29 16:45:01,644-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [42928c56] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 42928c56, Job ID: 4491f178-d135-496a-b5fc-c358a3fed938, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:45:01,648-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [42928c56] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 74748c0c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:45:01,650-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [42928c56] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 74748c0c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:45:01,661-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7222eac1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:45:01,662-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7222eac1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:50:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [1dd176a1] Autorecovering 2 hosts
>2018-05-29 16:50:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [1dd176a1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:50:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29e28c60] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:50:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29e28c60] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:50:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29e28c60] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:50:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29e28c60] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:50:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [29e28c60] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5fc0f2b5
>2018-05-29 16:50:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [29e28c60] FINISH, SetVdsStatusVDSCommand, log id: 5fc0f2b5
>2018-05-29 16:50:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29e28c60] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:50:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29e28c60] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:50:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [29e28c60] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:50:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [35bcce4e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:50:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [35bcce4e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:50:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [35bcce4e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:50:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [35bcce4e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:50:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [35bcce4e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2254df9e
>2018-05-29 16:50:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [35bcce4e] FINISH, SetVdsStatusVDSCommand, log id: 2254df9e
>2018-05-29 16:50:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [35bcce4e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 10a0306d
>2018-05-29 16:50:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [35bcce4e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 10a0306d
>2018-05-29 16:50:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [35bcce4e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:50:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [35bcce4e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:50:02,369-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6389d15d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 65385da4
>2018-05-29 16:50:02,371-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6389d15d] FINISH, GetHardwareInfoVDSCommand, log id: 65385da4
>2018-05-29 16:50:02,384-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [7d82e8a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:50:02,388-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [1e973eb4] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:50:02,391-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [7aeaf68] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5dbb17fa
>2018-05-29 16:50:02,393-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [7aeaf68] FINISH, GetHardwareInfoVDSCommand, log id: 5dbb17fa
>2018-05-29 16:50:02,394-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5084914f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:50:02,407-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [643de52a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:50:02,407-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5084914f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7b65bcd4
>2018-05-29 16:50:02,411-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [509975ac] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:50:02,416-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [77264588] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:50:02,423-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [77264588] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 78a7ee80
>2018-05-29 16:50:02,772-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5084914f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:50:02,772-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5084914f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7b65bcd4
>2018-05-29 16:50:02,776-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5084914f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:50:02,777-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5084914f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:50:02,780-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5084914f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:50:02,781-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5084914f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:50:02,785-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5084914f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:50:02,786-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5084914f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:50:02,789-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5084914f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:50:02,791-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5084914f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5084914f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:50:02,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 480d6cb8
>2018-05-29 16:50:02,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:50:02,798-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [77264588] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:50:02,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [77264588] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 78a7ee80
>2018-05-29 16:50:02,802-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [77264588] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:50:02,803-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [77264588] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:50:02,807-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [77264588] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:50:02,807-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [77264588] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:50:02,811-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [77264588] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:50:02,812-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [77264588] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:50:02,815-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [77264588] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:50:02,818-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [77264588] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 77264588, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:50:02,895-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:50:02,895-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:50:02,895-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:50:02,895-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:50:02,895-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 480d6cb8
>2018-05-29 16:50:02,895-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:50:02,901-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4868a953
>2018-05-29 16:50:02,901-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:50:02,909-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [20df8233] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:50:02,910-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [20df8233] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 656cc772
>2018-05-29 16:50:02,912-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [20df8233] FINISH, SetVdsStatusVDSCommand, log id: 656cc772
>2018-05-29 16:50:02,921-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [20df8233] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 20df8233, Job ID: f1ceb27b-5436-40b7-a84e-57e23c4f5638, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:50:02,925-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [20df8233] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1e973eb4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:50:02,928-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [20df8233] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1e973eb4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:50:02,939-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [38ba64d9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:50:02,940-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [38ba64d9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:50:02,997-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:50:02,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:50:02,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:50:02,997-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:50:02,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4868a953
>2018-05-29 16:50:02,997-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:50:03,010-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4ec673b0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:50:03,011-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4ec673b0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 769fef2
>2018-05-29 16:50:03,013-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4ec673b0] FINISH, SetVdsStatusVDSCommand, log id: 769fef2
>2018-05-29 16:50:03,019-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ec673b0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4ec673b0, Job ID: 443d13ad-15b9-4ee1-82cd-d6d8d022105c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:50:03,023-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ec673b0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 509975ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:50:03,026-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ec673b0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 509975ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:50:03,038-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [3a681f65] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:50:03,039-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [3a681f65] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:54:43,123-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler7) [1d4b1893] Attempting to update VMs/Templates Ovf.
>2018-05-29 16:54:43,126-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 16:54:43,126-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 16:54:43,126-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:54:43,131-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 16:54:43,135-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 16:54:43,135-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 16:54:43,135-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 16:54:43,135-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 16:54:43,136-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 16:54:43,136-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 16:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [257852a1] Autorecovering 2 hosts
>2018-05-29 16:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [257852a1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:55:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a343a87] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:55:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a343a87] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:55:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a343a87] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:55:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a343a87] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:55:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7a343a87] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 62b35175
>2018-05-29 16:55:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7a343a87] FINISH, SetVdsStatusVDSCommand, log id: 62b35175
>2018-05-29 16:55:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a343a87] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:55:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a343a87] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 16:55:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [7a343a87] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:55:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6178b23f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:55:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6178b23f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 16:55:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6178b23f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:55:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6178b23f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:55:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6178b23f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 454ae6aa
>2018-05-29 16:55:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6178b23f] FINISH, SetVdsStatusVDSCommand, log id: 454ae6aa
>2018-05-29 16:55:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [6178b23f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 205b20fa
>2018-05-29 16:55:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [6178b23f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 205b20fa
>2018-05-29 16:55:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6178b23f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 16:55:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6178b23f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 16:55:00,722-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [664583b0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6ccc6cff
>2018-05-29 16:55:00,723-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2b5cc194] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 67d3824a
>2018-05-29 16:55:00,724-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [664583b0] FINISH, GetHardwareInfoVDSCommand, log id: 6ccc6cff
>2018-05-29 16:55:00,725-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2b5cc194] FINISH, GetHardwareInfoVDSCommand, log id: 67d3824a
>2018-05-29 16:55:00,740-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [55419cfd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:55:00,742-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [527fcd88] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:55:00,750-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [45326a77] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:55:00,751-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [51c13625] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:55:00,756-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [30a9593c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:55:00,757-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [508b8f90] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 16:55:00,763-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [30a9593c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 478591ce
>2018-05-29 16:55:00,769-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [508b8f90] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 789b0442
>2018-05-29 16:55:01,139-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30a9593c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:55:01,139-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [30a9593c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 478591ce
>2018-05-29 16:55:01,143-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30a9593c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:55:01,144-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [30a9593c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:55:01,148-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30a9593c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:55:01,149-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [30a9593c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:55:01,151-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [508b8f90] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 16:55:01,151-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [508b8f90] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 789b0442
>2018-05-29 16:55:01,152-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30a9593c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:55:01,153-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [30a9593c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:55:01,155-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [508b8f90] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:55:01,155-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [508b8f90] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:55:01,156-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [30a9593c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:55:01,159-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30a9593c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 30a9593c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:55:01,160-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [508b8f90] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:55:01,161-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [508b8f90] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:55:01,162-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4d408bd
>2018-05-29 16:55:01,162-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:55:01,165-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [508b8f90] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 16:55:01,166-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [508b8f90] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 16:55:01,169-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [508b8f90] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 16:55:01,172-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [508b8f90] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 508b8f90, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 16:55:01,261-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:55:01,261-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:55:01,261-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:55:01,261-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:55:01,261-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4d408bd
>2018-05-29 16:55:01,261-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:55:01,263-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 766d15ab
>2018-05-29 16:55:01,263-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 16:55:01,274-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [4b0db259] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:55:01,275-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4b0db259] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: db515
>2018-05-29 16:55:01,277-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4b0db259] FINISH, SetVdsStatusVDSCommand, log id: db515
>2018-05-29 16:55:01,306-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4b0db259] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4b0db259, Job ID: 5ca51d00-4ebe-4a08-b15d-86735cd163aa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:55:01,310-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4b0db259] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 45326a77, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:55:01,312-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4b0db259] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 45326a77, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:55:01,323-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [2cd85270] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 16:55:01,324-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [2cd85270] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 16:55:01,362-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:55:01,362-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 16:55:01,362-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 16:55:01,362-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 16:55:01,362-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 766d15ab
>2018-05-29 16:55:01,362-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 16:55:01,374-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [6ebd8f2c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:55:01,375-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6ebd8f2c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 75dff24
>2018-05-29 16:55:01,377-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6ebd8f2c] FINISH, SetVdsStatusVDSCommand, log id: 75dff24
>2018-05-29 16:55:01,383-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6ebd8f2c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6ebd8f2c, Job ID: dcd9e8df-7296-4ab9-b1b0-375027643a1b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 16:55:01,387-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6ebd8f2c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 51c13625, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 16:55:01,389-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6ebd8f2c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 51c13625, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 16:55:01,400-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [1e9a34b5] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 16:55:01,401-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [1e9a34b5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:00:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [67792366] Autorecovering 2 hosts
>2018-05-29 17:00:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [67792366] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:00:00,013-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [c924a64] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:00:00,020-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [c924a64] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:00:00,020-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [c924a64] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:00:00,020-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [c924a64] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:00:00,022-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [c924a64] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 48a84318
>2018-05-29 17:00:00,026-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [c924a64] FINISH, SetVdsStatusVDSCommand, log id: 48a84318
>2018-05-29 17:00:00,027-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [c924a64] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:00:00,028-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [c924a64] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:00:00,028-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [c924a64] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:00:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6d8669c0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:00:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6d8669c0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:00:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6d8669c0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:00:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6d8669c0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:00:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6d8669c0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6b9f77f7
>2018-05-29 17:00:00,045-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6d8669c0] FINISH, SetVdsStatusVDSCommand, log id: 6b9f77f7
>2018-05-29 17:00:00,047-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [6d8669c0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 58bc8c75
>2018-05-29 17:00:00,049-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [6d8669c0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 58bc8c75
>2018-05-29 17:00:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6d8669c0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:00:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6d8669c0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:00:01,952-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [148e16ab] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1befa970
>2018-05-29 17:00:01,954-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [148e16ab] FINISH, GetHardwareInfoVDSCommand, log id: 1befa970
>2018-05-29 17:00:01,969-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [2ab56c7f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:00:01,973-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7933049e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 78241f86
>2018-05-29 17:00:01,975-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7933049e] FINISH, GetHardwareInfoVDSCommand, log id: 78241f86
>2018-05-29 17:00:01,978-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [342355ba] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:00:01,983-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [33beda13] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:00:01,990-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [33beda13] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4a0a5ba3
>2018-05-29 17:00:01,995-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [1783666c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:00:01,999-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [2e2f6d9f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:00:02,005-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [410905b9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:00:02,017-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [410905b9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 31cdcf83
>2018-05-29 17:00:02,390-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [410905b9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:00:02,390-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [410905b9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 31cdcf83
>2018-05-29 17:00:02,394-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [410905b9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:00:02,395-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [410905b9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:00:02,399-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [410905b9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:00:02,399-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [410905b9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:00:02,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [410905b9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:00:02,404-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [410905b9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:00:02,407-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [410905b9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:00:02,409-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [410905b9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 410905b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:00:02,412-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 32e670d3
>2018-05-29 17:00:02,412-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:00:02,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:00:02,511-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:00:02,511-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:00:02,511-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:00:02,511-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 32e670d3
>2018-05-29 17:00:02,511-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:00:02,524-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [287116d8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:00:02,525-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [287116d8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6c4b8f04
>2018-05-29 17:00:02,528-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [287116d8] FINISH, SetVdsStatusVDSCommand, log id: 6c4b8f04
>2018-05-29 17:00:02,561-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [287116d8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 287116d8, Job ID: 09283aa5-a2fe-4079-8a94-f62c85498139, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:00:02,565-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [287116d8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2e2f6d9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:00:02,568-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [287116d8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2e2f6d9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:00:02,578-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [78abae48] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:00:02,580-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [78abae48] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:00:03,356-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [33beda13] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:00:03,356-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [33beda13] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4a0a5ba3
>2018-05-29 17:00:03,360-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [33beda13] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:00:03,361-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [33beda13] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:00:03,365-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [33beda13] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:00:03,365-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [33beda13] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:00:03,369-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [33beda13] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:00:03,370-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [33beda13] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:00:03,373-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [33beda13] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:00:03,376-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [33beda13] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 33beda13, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:00:03,378-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4ccb30b4
>2018-05-29 17:00:03,378-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:00:03,475-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:00:03,475-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:00:03,475-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:00:03,475-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:00:03,475-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4ccb30b4
>2018-05-29 17:00:03,475-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:00:03,488-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [3915399e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:00:03,489-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3915399e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6c910db0
>2018-05-29 17:00:03,491-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3915399e] FINISH, SetVdsStatusVDSCommand, log id: 6c910db0
>2018-05-29 17:00:03,497-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3915399e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3915399e, Job ID: abc0a669-8a56-40f4-b275-fbce49505708, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:00:03,501-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3915399e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 342355ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:00:03,503-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3915399e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 342355ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:00:03,515-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [6f64556] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:00:03,516-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [6f64556] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [55d06484] Autorecovering 2 hosts
>2018-05-29 17:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [55d06484] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:05:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [49638486] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:05:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [49638486] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:05:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [49638486] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:05:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [49638486] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:05:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [49638486] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c8ab1c1
>2018-05-29 17:05:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [49638486] FINISH, SetVdsStatusVDSCommand, log id: 7c8ab1c1
>2018-05-29 17:05:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [49638486] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:05:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [49638486] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:05:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [49638486] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:05:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [70659e6a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:05:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [70659e6a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:05:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [70659e6a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:05:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [70659e6a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:05:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [70659e6a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ca5d426
>2018-05-29 17:05:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [70659e6a] FINISH, SetVdsStatusVDSCommand, log id: 2ca5d426
>2018-05-29 17:05:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [70659e6a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7e8ca2d4
>2018-05-29 17:05:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [70659e6a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7e8ca2d4
>2018-05-29 17:05:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [70659e6a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:05:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [70659e6a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:05:01,170-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [495250cb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1182974a
>2018-05-29 17:05:01,172-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [495250cb] FINISH, GetHardwareInfoVDSCommand, log id: 1182974a
>2018-05-29 17:05:01,186-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [34464424] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:05:01,195-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [306cfa28] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:05:01,201-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [51a7edbf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:05:01,257-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [51a7edbf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 406006e7
>2018-05-29 17:05:01,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [51a7edbf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:05:01,631-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [51a7edbf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 406006e7
>2018-05-29 17:05:01,635-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [51a7edbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:05:01,636-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [51a7edbf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:05:01,640-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [51a7edbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:05:01,641-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [51a7edbf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:05:01,645-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [51a7edbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:05:01,645-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [51a7edbf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:05:01,649-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [51a7edbf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:05:01,651-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [51a7edbf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 51a7edbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:05:01,658-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3a25327d
>2018-05-29 17:05:01,658-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:05:01,757-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:05:01,757-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:05:01,757-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:05:01,757-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:05:01,757-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3a25327d
>2018-05-29 17:05:01,757-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:05:01,770-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4a99bd47] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:05:01,771-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4a99bd47] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43ac1d14
>2018-05-29 17:05:01,774-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4a99bd47] FINISH, SetVdsStatusVDSCommand, log id: 43ac1d14
>2018-05-29 17:05:01,805-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4a99bd47] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4a99bd47, Job ID: 0b254bb3-e56c-4fed-b957-ed70c47a2597, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:05:01,810-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4a99bd47] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 306cfa28, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:05:01,812-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4a99bd47] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 306cfa28, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:05:01,823-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [2d72159] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:05:01,824-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [2d72159] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:05:03,167-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2f7b853b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 476873fb
>2018-05-29 17:05:03,169-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2f7b853b] FINISH, GetHardwareInfoVDSCommand, log id: 476873fb
>2018-05-29 17:05:03,182-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3c440e13] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:05:03,187-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [7059e04a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:05:03,192-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [37c9ccf9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:05:03,200-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [37c9ccf9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 39e10eac
>2018-05-29 17:05:03,592-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [37c9ccf9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:05:03,592-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [37c9ccf9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 39e10eac
>2018-05-29 17:05:03,596-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [37c9ccf9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:05:03,597-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [37c9ccf9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:05:03,601-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [37c9ccf9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:05:03,601-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [37c9ccf9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:05:03,605-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [37c9ccf9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:05:03,606-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [37c9ccf9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:05:03,609-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [37c9ccf9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:05:03,612-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [37c9ccf9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 37c9ccf9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:05:03,618-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3eaea35
>2018-05-29 17:05:03,618-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:05:03,717-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:05:03,717-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:05:03,717-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:05:03,717-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:05:03,717-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3eaea35
>2018-05-29 17:05:03,717-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:05:03,730-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [27475e5e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:05:03,731-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [27475e5e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 425a98dc
>2018-05-29 17:05:03,733-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [27475e5e] FINISH, SetVdsStatusVDSCommand, log id: 425a98dc
>2018-05-29 17:05:03,766-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27475e5e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 27475e5e, Job ID: ada62e36-babc-461c-ab59-cc1a394d3e30, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:05:03,771-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27475e5e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7059e04a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:05:03,773-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27475e5e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7059e04a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:05:03,784-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [3b9dc3f1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:05:03,785-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [3b9dc3f1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3910ca68] Autorecovering 2 hosts
>2018-05-29 17:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3910ca68] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:10:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [72700d72] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:10:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [72700d72] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:10:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [72700d72] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:10:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [72700d72] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:10:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [72700d72] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 48354030
>2018-05-29 17:10:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [72700d72] FINISH, SetVdsStatusVDSCommand, log id: 48354030
>2018-05-29 17:10:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [72700d72] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:10:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [72700d72] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:10:00,073-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [72700d72] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:10:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [13d8a87a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:10:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [13d8a87a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:10:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [13d8a87a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:10:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [13d8a87a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:10:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [13d8a87a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6fcbe42
>2018-05-29 17:10:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [13d8a87a] FINISH, SetVdsStatusVDSCommand, log id: 6fcbe42
>2018-05-29 17:10:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [13d8a87a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5ff1eab9
>2018-05-29 17:10:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [13d8a87a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5ff1eab9
>2018-05-29 17:10:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [13d8a87a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:10:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [13d8a87a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:10:01,358-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [632f1043] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1f34f211
>2018-05-29 17:10:01,359-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [632f1043] FINISH, GetHardwareInfoVDSCommand, log id: 1f34f211
>2018-05-29 17:10:01,372-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [6a959407] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:10:01,381-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [3ce63245] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:10:01,386-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6852eed9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:10:01,392-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6852eed9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 488c1289
>2018-05-29 17:10:01,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6852eed9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:10:01,773-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6852eed9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 488c1289
>2018-05-29 17:10:01,777-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6852eed9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:10:01,778-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6852eed9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:10:01,782-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6852eed9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:10:01,782-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6852eed9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:10:01,786-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6852eed9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:10:01,787-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6852eed9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:10:01,790-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6852eed9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:10:01,793-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6852eed9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6852eed9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:10:01,796-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4c23b5c9
>2018-05-29 17:10:01,796-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:10:01,893-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:10:01,893-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:10:01,893-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:10:01,893-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:10:01,893-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4c23b5c9
>2018-05-29 17:10:01,893-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:10:01,907-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [168399fe] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:10:01,909-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [168399fe] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 580b1f14
>2018-05-29 17:10:01,911-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [168399fe] FINISH, SetVdsStatusVDSCommand, log id: 580b1f14
>2018-05-29 17:10:01,942-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [168399fe] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 168399fe, Job ID: 520ebbbb-4562-4172-bae7-1a8391c443e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:10:01,947-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [168399fe] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3ce63245, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:10:01,949-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [168399fe] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3ce63245, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:10:01,960-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [4cc43fe8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:10:01,961-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [4cc43fe8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:10:02,684-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [343f98b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 54c1e6c4
>2018-05-29 17:10:02,686-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [343f98b] FINISH, GetHardwareInfoVDSCommand, log id: 54c1e6c4
>2018-05-29 17:10:02,698-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [70afd45f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:10:02,703-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [2cfe1494] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:10:02,708-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [4072a2e2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:10:02,715-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [4072a2e2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 69fc102f
>2018-05-29 17:10:03,088-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4072a2e2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:10:03,088-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [4072a2e2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 69fc102f
>2018-05-29 17:10:03,092-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4072a2e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:10:03,093-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4072a2e2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:10:03,097-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4072a2e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:10:03,097-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4072a2e2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:10:03,102-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4072a2e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:10:03,102-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4072a2e2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:10:03,106-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [4072a2e2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:10:03,109-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4072a2e2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4072a2e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:10:03,112-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 71e7282b
>2018-05-29 17:10:03,112-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:10:03,208-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:10:03,208-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:10:03,208-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:10:03,208-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:10:03,208-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 71e7282b
>2018-05-29 17:10:03,208-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:10:03,223-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [eef516e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:10:03,224-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [eef516e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f7091dd
>2018-05-29 17:10:03,226-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [eef516e] FINISH, SetVdsStatusVDSCommand, log id: 1f7091dd
>2018-05-29 17:10:03,232-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [eef516e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: eef516e, Job ID: b779860d-1cf8-47de-b77f-87f559776da7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:10:03,237-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [eef516e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2cfe1494, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:10:03,239-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [eef516e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2cfe1494, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:10:03,250-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [4996ccc7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:10:03,251-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4996ccc7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:15:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [21d7e152] Autorecovering 2 hosts
>2018-05-29 17:15:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [21d7e152] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:15:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [19cf5177] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:15:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [19cf5177] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:15:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [19cf5177] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:15:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [19cf5177] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:15:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [19cf5177] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3de7be04
>2018-05-29 17:15:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [19cf5177] FINISH, SetVdsStatusVDSCommand, log id: 3de7be04
>2018-05-29 17:15:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [19cf5177] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:15:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [19cf5177] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:15:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [19cf5177] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:15:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7029d6de] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7029d6de] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7029d6de] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7029d6de] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:15:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7029d6de] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d76dff4
>2018-05-29 17:15:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7029d6de] FINISH, SetVdsStatusVDSCommand, log id: 2d76dff4
>2018-05-29 17:15:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7029d6de] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 248d2415
>2018-05-29 17:15:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7029d6de] FINISH, SetHaMaintenanceModeVDSCommand, log id: 248d2415
>2018-05-29 17:15:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7029d6de] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:15:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7029d6de] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:15:00,943-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [38dc62de] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: c152f47
>2018-05-29 17:15:00,945-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [38dc62de] FINISH, GetHardwareInfoVDSCommand, log id: c152f47
>2018-05-29 17:15:00,958-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2997745b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:15:00,967-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [2da2e598] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:15:00,973-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [627e752] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:15:00,985-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [627e752] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1c503ce9
>2018-05-29 17:15:01,355-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [627e752] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:15:01,355-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [627e752] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1c503ce9
>2018-05-29 17:15:01,359-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [627e752] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:15:01,359-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [627e752] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:15:01,364-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [627e752] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:15:01,364-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [627e752] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:15:01,368-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [627e752] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:15:01,369-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [627e752] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:15:01,372-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [627e752] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:15:01,374-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [627e752] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 627e752, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:15:01,381-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 632098d1
>2018-05-29 17:15:01,381-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:15:01,480-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:15:01,480-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:15:01,480-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:15:01,480-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:15:01,480-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 632098d1
>2018-05-29 17:15:01,480-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:15:01,494-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [5ea139f3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:15:01,495-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5ea139f3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43945540
>2018-05-29 17:15:01,498-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5ea139f3] FINISH, SetVdsStatusVDSCommand, log id: 43945540
>2018-05-29 17:15:01,530-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5ea139f3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5ea139f3, Job ID: 1e5ddabb-5c7c-46d3-af44-11ad3f276464, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:15:01,535-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5ea139f3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2da2e598, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:15:01,537-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5ea139f3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2da2e598, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:15:01,548-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [11ce3fb2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:15:01,549-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [11ce3fb2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:15:02,571-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2cb0d106] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 25bc382
>2018-05-29 17:15:02,573-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2cb0d106] FINISH, GetHardwareInfoVDSCommand, log id: 25bc382
>2018-05-29 17:15:02,586-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [4c4b9cd6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:15:02,590-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [74e99cf0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:15:02,596-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3e0aa180] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:15:02,602-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3e0aa180] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 34c8ff17
>2018-05-29 17:15:02,978-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e0aa180] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:15:02,978-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3e0aa180] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 34c8ff17
>2018-05-29 17:15:02,982-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e0aa180] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:15:02,983-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e0aa180] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:15:02,986-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e0aa180] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:15:02,987-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e0aa180] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:15:02,991-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e0aa180] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:15:02,992-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e0aa180] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:15:02,995-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3e0aa180] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:15:02,997-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e0aa180] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3e0aa180, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:15:03,000-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5e34dc99
>2018-05-29 17:15:03,000-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:15:03,097-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:15:03,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:15:03,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:15:03,097-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:15:03,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5e34dc99
>2018-05-29 17:15:03,097-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:15:03,111-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [3c88f790] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:15:03,112-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3c88f790] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43574dc8
>2018-05-29 17:15:03,114-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3c88f790] FINISH, SetVdsStatusVDSCommand, log id: 43574dc8
>2018-05-29 17:15:03,120-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c88f790] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c88f790, Job ID: f38d0680-08bb-456b-a09f-7c59b31bcbd8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:15:03,125-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c88f790] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 74e99cf0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:15:03,127-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c88f790] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 74e99cf0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:15:03,138-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [747279f7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:15:03,139-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [747279f7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [51ffeec0] Autorecovering 2 hosts
>2018-05-29 17:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [51ffeec0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:20:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3e4dea9a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:20:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3e4dea9a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:20:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3e4dea9a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:20:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3e4dea9a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:20:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3e4dea9a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5c2ca86c
>2018-05-29 17:20:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3e4dea9a] FINISH, SetVdsStatusVDSCommand, log id: 5c2ca86c
>2018-05-29 17:20:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3e4dea9a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:20:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3e4dea9a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:20:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3e4dea9a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:20:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [118f1326] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:20:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [118f1326] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:20:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [118f1326] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:20:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [118f1326] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:20:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [118f1326] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4570c734
>2018-05-29 17:20:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [118f1326] FINISH, SetVdsStatusVDSCommand, log id: 4570c734
>2018-05-29 17:20:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [118f1326] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4af9f2ae
>2018-05-29 17:20:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [118f1326] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4af9f2ae
>2018-05-29 17:20:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [118f1326] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:20:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [118f1326] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:20:00,751-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6263f6f4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: f4a9cf2
>2018-05-29 17:20:00,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6263f6f4] FINISH, GetHardwareInfoVDSCommand, log id: f4a9cf2
>2018-05-29 17:20:00,765-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7b348628] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:20:00,773-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3f3da94e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:20:00,779-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [372ffecd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:20:00,791-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [372ffecd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 625201a
>2018-05-29 17:20:01,171-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [372ffecd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:20:01,171-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [372ffecd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 625201a
>2018-05-29 17:20:01,175-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [372ffecd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:20:01,176-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [372ffecd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:20:01,180-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [372ffecd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:20:01,180-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [372ffecd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:20:01,184-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [372ffecd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:20:01,185-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [372ffecd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:20:01,188-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [372ffecd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:20:01,191-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [372ffecd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 372ffecd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:20:01,197-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 27f5b8b0
>2018-05-29 17:20:01,197-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:20:01,292-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:20:01,292-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:20:01,292-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:20:01,293-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:20:01,293-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 27f5b8b0
>2018-05-29 17:20:01,293-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:20:01,306-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [75d3e20a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:20:01,307-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [75d3e20a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 79e0826
>2018-05-29 17:20:01,310-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [75d3e20a] FINISH, SetVdsStatusVDSCommand, log id: 79e0826
>2018-05-29 17:20:01,341-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [75d3e20a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 75d3e20a, Job ID: cb408361-699f-461d-b984-319f5e0618b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:20:01,346-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [75d3e20a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3f3da94e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:20:01,348-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [75d3e20a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3f3da94e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:20:01,360-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [2b13373b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:20:01,361-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [2b13373b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:20:02,270-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [89c4559] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 644c721a
>2018-05-29 17:20:02,272-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [89c4559] FINISH, GetHardwareInfoVDSCommand, log id: 644c721a
>2018-05-29 17:20:02,286-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [77b0439] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:20:02,294-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [f85bef9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:20:02,300-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2c378018] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:20:02,312-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2c378018] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23df10a9
>2018-05-29 17:20:03,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2c378018] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:20:03,680-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2c378018] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 23df10a9
>2018-05-29 17:20:03,684-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2c378018] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:20:03,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2c378018] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:20:03,689-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2c378018] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:20:03,689-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2c378018] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:20:03,693-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2c378018] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:20:03,694-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2c378018] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:20:03,697-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2c378018] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:20:03,700-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2c378018] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2c378018, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:20:03,707-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 755bde48
>2018-05-29 17:20:03,707-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:20:03,806-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:20:03,806-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:20:03,806-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:20:03,806-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:20:03,806-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 755bde48
>2018-05-29 17:20:03,806-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:20:03,819-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1d21f687] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:20:03,821-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1d21f687] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13aa60e7
>2018-05-29 17:20:03,823-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1d21f687] FINISH, SetVdsStatusVDSCommand, log id: 13aa60e7
>2018-05-29 17:20:03,854-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1d21f687] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1d21f687, Job ID: 1a07df34-a82e-4fc9-8cbc-85fb158a43fd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:20:03,859-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1d21f687] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: f85bef9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:20:03,861-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1d21f687] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: f85bef9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:20:03,872-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [6f004968] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:20:03,873-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [6f004968] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:25:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2c21f8fe] Autorecovering 2 hosts
>2018-05-29 17:25:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2c21f8fe] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:25:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [79e1075d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:25:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [79e1075d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:25:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [79e1075d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:25:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [79e1075d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:25:00,046-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [79e1075d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5ae92b1f
>2018-05-29 17:25:00,049-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [79e1075d] FINISH, SetVdsStatusVDSCommand, log id: 5ae92b1f
>2018-05-29 17:25:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [79e1075d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:25:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [79e1075d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:25:00,052-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [79e1075d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:25:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [272c1f21] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:25:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [272c1f21] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:25:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [272c1f21] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:25:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [272c1f21] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:25:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [272c1f21] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54725e9f
>2018-05-29 17:25:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [272c1f21] FINISH, SetVdsStatusVDSCommand, log id: 54725e9f
>2018-05-29 17:25:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [272c1f21] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7cd9eafd
>2018-05-29 17:25:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [272c1f21] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7cd9eafd
>2018-05-29 17:25:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [272c1f21] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:25:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [272c1f21] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:25:01,555-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4d19658c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: d4d03ab
>2018-05-29 17:25:01,556-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4d19658c] FINISH, GetHardwareInfoVDSCommand, log id: d4d03ab
>2018-05-29 17:25:01,570-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [36614150] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:25:01,574-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [27f1cfb6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:25:01,579-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6dae818f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:25:01,586-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6dae818f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 31ead308
>2018-05-29 17:25:01,960-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6dae818f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:25:01,960-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6dae818f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 31ead308
>2018-05-29 17:25:01,961-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [63cb55ea] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 53049b21
>2018-05-29 17:25:01,963-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [63cb55ea] FINISH, GetHardwareInfoVDSCommand, log id: 53049b21
>2018-05-29 17:25:01,964-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6dae818f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:25:01,965-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6dae818f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:25:01,969-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6dae818f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:25:01,970-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6dae818f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:25:01,974-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6dae818f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:25:01,975-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6dae818f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:25:01,976-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [903f75f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:25:01,979-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6dae818f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:25:01,980-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5dc66ca7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:25:01,982-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6dae818f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6dae818f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:25:01,985-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5a8bd1d2
>2018-05-29 17:25:01,985-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:25:01,986-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [78cba89e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:25:01,993-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [78cba89e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5737ac7f
>2018-05-29 17:25:02,083-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:25:02,083-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:25:02,083-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:25:02,083-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:25:02,083-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5a8bd1d2
>2018-05-29 17:25:02,083-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:25:02,097-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [550aefcd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:25:02,098-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [550aefcd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54d2bd55
>2018-05-29 17:25:02,100-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [550aefcd] FINISH, SetVdsStatusVDSCommand, log id: 54d2bd55
>2018-05-29 17:25:02,132-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [550aefcd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 550aefcd, Job ID: 91782706-c361-46e8-889d-dfc64038f7e9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:25:02,136-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [550aefcd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 27f1cfb6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:25:02,138-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [550aefcd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 27f1cfb6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:25:02,149-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [2205ac88] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:25:02,150-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [2205ac88] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:25:02,369-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [78cba89e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:25:02,369-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [78cba89e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5737ac7f
>2018-05-29 17:25:02,374-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [78cba89e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:25:02,374-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [78cba89e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:25:02,378-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [78cba89e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:25:02,379-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [78cba89e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:25:02,383-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [78cba89e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:25:02,383-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [78cba89e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:25:02,386-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [78cba89e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:25:02,389-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [78cba89e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 78cba89e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:25:02,392-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1fe4e5d
>2018-05-29 17:25:02,392-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:25:02,490-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:25:02,490-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:25:02,490-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:25:02,490-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:25:02,490-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1fe4e5d
>2018-05-29 17:25:02,490-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:25:02,503-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [5b84f71f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:25:02,504-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5b84f71f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 719d905a
>2018-05-29 17:25:02,506-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5b84f71f] FINISH, SetVdsStatusVDSCommand, log id: 719d905a
>2018-05-29 17:25:02,512-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5b84f71f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5b84f71f, Job ID: 9f9f56a8-0868-4864-af95-864e462ee682, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:25:02,516-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5b84f71f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5dc66ca7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:25:02,519-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5b84f71f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5dc66ca7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:25:02,530-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [661c1fab] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:25:02,531-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [661c1fab] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:30:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [532ef1f7] Autorecovering 2 hosts
>2018-05-29 17:30:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [532ef1f7] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:30:00,031-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7fafbd72] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:30:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7fafbd72] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:30:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7fafbd72] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:30:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7fafbd72] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:30:00,039-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7fafbd72] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5bf7337e
>2018-05-29 17:30:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7fafbd72] FINISH, SetVdsStatusVDSCommand, log id: 5bf7337e
>2018-05-29 17:30:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7fafbd72] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:30:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7fafbd72] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:30:00,045-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7fafbd72] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:30:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6a413878] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:30:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6a413878] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:30:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6a413878] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:30:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6a413878] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:30:00,058-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6a413878] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 749d6366
>2018-05-29 17:30:00,060-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6a413878] FINISH, SetVdsStatusVDSCommand, log id: 749d6366
>2018-05-29 17:30:00,062-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6a413878] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1082beb4
>2018-05-29 17:30:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6a413878] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1082beb4
>2018-05-29 17:30:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6a413878] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:30:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6a413878] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:30:02,809-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [1b09a1a9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 240ddbd2
>2018-05-29 17:30:02,811-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [1b09a1a9] FINISH, GetHardwareInfoVDSCommand, log id: 240ddbd2
>2018-05-29 17:30:02,825-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [53df450a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:30:02,829-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [7eaec964] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:30:02,835-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7ff0b433] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:30:02,847-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7ff0b433] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6ab85a9c
>2018-05-29 17:30:03,148-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1d0b17e7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: fa6389a
>2018-05-29 17:30:03,150-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1d0b17e7] FINISH, GetHardwareInfoVDSCommand, log id: fa6389a
>2018-05-29 17:30:03,163-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [20425622] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:30:03,167-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [2b17de51] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:30:03,173-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5e33e3cc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:30:03,180-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5e33e3cc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 422930ef
>2018-05-29 17:30:03,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5e33e3cc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:30:03,566-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5e33e3cc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 422930ef
>2018-05-29 17:30:03,570-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5e33e3cc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:30:03,571-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5e33e3cc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:30:03,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5e33e3cc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:30:03,576-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5e33e3cc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:30:03,580-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5e33e3cc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:30:03,580-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5e33e3cc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:30:03,628-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5e33e3cc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:30:03,631-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5e33e3cc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5e33e3cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:30:03,638-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7fbc806b
>2018-05-29 17:30:03,638-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:30:03,736-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:30:03,736-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:30:03,736-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:30:03,736-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:30:03,736-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7fbc806b
>2018-05-29 17:30:03,736-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:30:03,750-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [39c43a5b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:30:03,751-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [39c43a5b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: dd4bb1f
>2018-05-29 17:30:03,753-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [39c43a5b] FINISH, SetVdsStatusVDSCommand, log id: dd4bb1f
>2018-05-29 17:30:03,786-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39c43a5b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 39c43a5b, Job ID: 40ba19ca-1cce-48c3-9382-eeef71ff4d11, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:30:03,791-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39c43a5b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2b17de51, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:30:03,793-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39c43a5b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2b17de51, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:30:03,808-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [48401a59] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:30:03,809-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [48401a59] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:30:04,219-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7ff0b433] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:30:04,219-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7ff0b433] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6ab85a9c
>2018-05-29 17:30:04,223-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7ff0b433] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:30:04,224-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7ff0b433] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:30:04,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7ff0b433] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:30:04,229-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7ff0b433] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:30:04,233-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7ff0b433] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:30:04,234-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7ff0b433] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:30:04,237-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7ff0b433] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:30:04,240-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7ff0b433] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7ff0b433, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:30:04,243-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 824a683
>2018-05-29 17:30:04,243-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:30:04,340-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:30:04,340-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:30:04,340-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:30:04,340-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:30:04,340-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 824a683
>2018-05-29 17:30:04,340-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:30:04,353-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4f995eec] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:30:04,354-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4f995eec] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4560e92c
>2018-05-29 17:30:04,356-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4f995eec] FINISH, SetVdsStatusVDSCommand, log id: 4560e92c
>2018-05-29 17:30:04,362-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f995eec] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4f995eec, Job ID: 5b1b2175-c620-4fde-bcb5-2773e47fbbab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:30:04,367-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f995eec] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7eaec964, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:30:04,369-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f995eec] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7eaec964, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:30:04,380-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [3e0e2f15] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:30:04,381-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [3e0e2f15] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2c514688] Autorecovering 2 hosts
>2018-05-29 17:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2c514688] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:35:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3ac65401] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:35:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3ac65401] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:35:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3ac65401] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:35:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3ac65401] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:35:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3ac65401] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7379ae95
>2018-05-29 17:35:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3ac65401] FINISH, SetVdsStatusVDSCommand, log id: 7379ae95
>2018-05-29 17:35:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3ac65401] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:35:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3ac65401] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:35:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [3ac65401] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:35:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7e61b152] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:35:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7e61b152] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:35:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7e61b152] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:35:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7e61b152] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:35:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7e61b152] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30d032c8
>2018-05-29 17:35:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7e61b152] FINISH, SetVdsStatusVDSCommand, log id: 30d032c8
>2018-05-29 17:35:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [7e61b152] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7fb8e440
>2018-05-29 17:35:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [7e61b152] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7fb8e440
>2018-05-29 17:35:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7e61b152] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:35:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7e61b152] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:35:01,490-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [52c02ccf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 324556c8
>2018-05-29 17:35:01,492-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [52c02ccf] FINISH, GetHardwareInfoVDSCommand, log id: 324556c8
>2018-05-29 17:35:01,504-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [192078e2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:35:01,512-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [6b3ac934] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:35:01,517-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5e3a7f25] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:35:01,528-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5e3a7f25] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5957bcfa
>2018-05-29 17:35:01,903-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5e3a7f25] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:35:01,903-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5e3a7f25] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5957bcfa
>2018-05-29 17:35:01,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5e3a7f25] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:35:01,908-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5e3a7f25] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:35:01,911-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5e3a7f25] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:35:01,912-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5e3a7f25] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:35:01,916-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5e3a7f25] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:35:01,916-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5e3a7f25] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:35:01,919-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5e3a7f25] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:35:01,922-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5e3a7f25] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5e3a7f25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:35:01,928-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 48074259
>2018-05-29 17:35:01,928-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:35:01,998-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2d33a2b7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 70dee6c0
>2018-05-29 17:35:01,999-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2d33a2b7] FINISH, GetHardwareInfoVDSCommand, log id: 70dee6c0
>2018-05-29 17:35:02,011-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [64af1bbf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:35:02,016-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [63f6a33] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:35:02,021-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6d225339] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:35:02,024-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:35:02,024-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:35:02,024-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:35:02,024-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:35:02,024-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 48074259
>2018-05-29 17:35:02,024-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:35:02,036-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6d225339] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 41e27c62
>2018-05-29 17:35:02,041-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [6bce060a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:35:02,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6bce060a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 301f275f
>2018-05-29 17:35:02,045-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6bce060a] FINISH, SetVdsStatusVDSCommand, log id: 301f275f
>2018-05-29 17:35:02,076-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6bce060a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6bce060a, Job ID: 4c2655b9-6a94-4498-a5d3-f0117b6d4896, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:35:02,081-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6bce060a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6b3ac934, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:35:02,083-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6bce060a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6b3ac934, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:35:02,094-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [110dde19] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:35:02,095-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [110dde19] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:35:02,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6d225339] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:35:02,401-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6d225339] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 41e27c62
>2018-05-29 17:35:02,405-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6d225339] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:35:02,406-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6d225339] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:35:02,409-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6d225339] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:35:02,410-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6d225339] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:35:02,414-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6d225339] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:35:02,415-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6d225339] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:35:02,418-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6d225339] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:35:02,420-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6d225339] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6d225339, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:35:02,423-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4f6a6a6e
>2018-05-29 17:35:02,423-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:35:02,523-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:35:02,523-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:35:02,523-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:35:02,523-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:35:02,523-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4f6a6a6e
>2018-05-29 17:35:02,523-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:35:02,536-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [757662af] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:35:02,538-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [757662af] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 370ca352
>2018-05-29 17:35:02,540-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [757662af] FINISH, SetVdsStatusVDSCommand, log id: 370ca352
>2018-05-29 17:35:02,545-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [757662af] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 757662af, Job ID: 496932b1-8ed7-4cbe-8cd1-0d06375a52e9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:35:02,550-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [757662af] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 63f6a33, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:35:02,552-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [757662af] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 63f6a33, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:35:02,563-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [17c90764] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:35:02,564-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [17c90764] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:40:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [4758d012] Autorecovering 2 hosts
>2018-05-29 17:40:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [4758d012] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:40:00,036-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c845482] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:40:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c845482] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:40:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c845482] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:40:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c845482] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:40:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6c845482] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 44fdcb31
>2018-05-29 17:40:00,047-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6c845482] FINISH, SetVdsStatusVDSCommand, log id: 44fdcb31
>2018-05-29 17:40:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c845482] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:40:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c845482] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:40:00,050-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6c845482] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:40:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [187b6309] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:40:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [187b6309] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:40:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [187b6309] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:40:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [187b6309] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:40:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [187b6309] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2df88070
>2018-05-29 17:40:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [187b6309] FINISH, SetVdsStatusVDSCommand, log id: 2df88070
>2018-05-29 17:40:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [187b6309] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 51ab559a
>2018-05-29 17:40:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [187b6309] FINISH, SetHaMaintenanceModeVDSCommand, log id: 51ab559a
>2018-05-29 17:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [187b6309] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [187b6309] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:40:00,217-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [8fae41c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 71457182
>2018-05-29 17:40:00,219-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [8fae41c] FINISH, GetHardwareInfoVDSCommand, log id: 71457182
>2018-05-29 17:40:00,232-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [669b945] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:40:00,240-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [cfcf71e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:40:00,245-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [28e912cf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:40:00,256-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [28e912cf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1b3a6b3a
>2018-05-29 17:40:00,630-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [28e912cf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:40:00,630-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [28e912cf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1b3a6b3a
>2018-05-29 17:40:00,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [28e912cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:40:00,634-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [28e912cf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:40:00,638-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [28e912cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:40:00,639-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [28e912cf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:40:00,643-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [28e912cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:40:00,643-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [28e912cf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:40:00,646-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [28e912cf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:40:00,649-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [28e912cf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 28e912cf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:40:00,655-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 9d5f80
>2018-05-29 17:40:00,655-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:40:00,752-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:40:00,752-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:40:00,752-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:40:00,752-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:40:00,752-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 9d5f80
>2018-05-29 17:40:00,752-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:40:00,767-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [256e432b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:40:00,768-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [256e432b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5968ed04
>2018-05-29 17:40:00,770-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [256e432b] FINISH, SetVdsStatusVDSCommand, log id: 5968ed04
>2018-05-29 17:40:00,802-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [256e432b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 256e432b, Job ID: c3c265ed-fdda-4cff-9146-b3a3218927f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:40:00,807-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [256e432b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: cfcf71e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:40:00,809-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [256e432b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: cfcf71e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:40:00,820-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [281dfe9d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:40:00,821-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [281dfe9d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:40:02,835-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [202766c4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5c81ac6b
>2018-05-29 17:40:02,837-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [202766c4] FINISH, GetHardwareInfoVDSCommand, log id: 5c81ac6b
>2018-05-29 17:40:02,850-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [52f162f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:40:02,855-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [249e7524] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:40:02,860-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [29384491] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:40:02,867-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [29384491] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 69908fa6
>2018-05-29 17:40:03,243-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [29384491] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:40:03,243-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [29384491] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 69908fa6
>2018-05-29 17:40:03,247-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [29384491] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:40:03,248-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [29384491] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:40:03,252-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [29384491] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:40:03,253-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [29384491] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:40:03,257-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [29384491] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:40:03,257-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [29384491] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:40:03,260-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [29384491] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:40:03,263-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [29384491] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 29384491, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:40:03,266-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4b030275
>2018-05-29 17:40:03,266-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:40:03,363-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:40:03,363-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:40:03,363-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:40:03,363-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:40:03,363-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4b030275
>2018-05-29 17:40:03,363-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:40:03,376-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [2df7baee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:40:03,377-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2df7baee] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6f4ee420
>2018-05-29 17:40:03,379-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2df7baee] FINISH, SetVdsStatusVDSCommand, log id: 6f4ee420
>2018-05-29 17:40:03,385-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2df7baee] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2df7baee, Job ID: b785c13d-5352-4084-8eaf-61a49bb92d28, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:40:03,390-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2df7baee] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 249e7524, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:40:03,392-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2df7baee] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 249e7524, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:40:03,402-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [3a1f5a0e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:40:03,403-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [3a1f5a0e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [65bcc42] Autorecovering 2 hosts
>2018-05-29 17:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [65bcc42] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:45:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2e33be16] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:45:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2e33be16] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:45:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2e33be16] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:45:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2e33be16] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:45:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2e33be16] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18fde54d
>2018-05-29 17:45:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2e33be16] FINISH, SetVdsStatusVDSCommand, log id: 18fde54d
>2018-05-29 17:45:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2e33be16] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:45:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2e33be16] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:45:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [2e33be16] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:45:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [e82e4c6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:45:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [e82e4c6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:45:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [e82e4c6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:45:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [e82e4c6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:45:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [e82e4c6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47002d6a
>2018-05-29 17:45:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [e82e4c6] FINISH, SetVdsStatusVDSCommand, log id: 47002d6a
>2018-05-29 17:45:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [e82e4c6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4e566fe1
>2018-05-29 17:45:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [e82e4c6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4e566fe1
>2018-05-29 17:45:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [e82e4c6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:45:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [e82e4c6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:45:01,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [57deba71] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 50610bf4
>2018-05-29 17:45:01,078-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [57deba71] FINISH, GetHardwareInfoVDSCommand, log id: 50610bf4
>2018-05-29 17:45:01,091-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [440ec4b8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:45:01,100-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [f6c57a9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:45:01,105-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [718fb07f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:45:01,116-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [718fb07f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 54457eed
>2018-05-29 17:45:01,427-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [706f5176] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 581e045a
>2018-05-29 17:45:01,429-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [706f5176] FINISH, GetHardwareInfoVDSCommand, log id: 581e045a
>2018-05-29 17:45:01,441-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [30b203aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:45:01,446-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [1100b670] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:45:01,451-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4b3a84eb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:45:01,457-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4b3a84eb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: b99d405
>2018-05-29 17:45:01,500-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [718fb07f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:45:01,500-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [718fb07f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 54457eed
>2018-05-29 17:45:01,504-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [718fb07f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:45:01,504-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [718fb07f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:45:01,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [718fb07f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:45:01,509-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [718fb07f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:45:01,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [718fb07f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:45:01,513-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [718fb07f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:45:01,516-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [718fb07f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:45:01,518-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [718fb07f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 718fb07f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:45:01,525-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7574f3b9
>2018-05-29 17:45:01,525-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:45:01,624-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:45:01,624-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:45:01,624-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:45:01,624-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:45:01,624-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7574f3b9
>2018-05-29 17:45:01,624-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:45:01,637-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [32d4ff2c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:45:01,639-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [32d4ff2c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 11b2e722
>2018-05-29 17:45:01,641-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [32d4ff2c] FINISH, SetVdsStatusVDSCommand, log id: 11b2e722
>2018-05-29 17:45:01,674-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [32d4ff2c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 32d4ff2c, Job ID: 5b805d7a-7dca-42a6-9074-d1b6b932cb5b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:45:01,678-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [32d4ff2c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: f6c57a9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:45:01,681-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [32d4ff2c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: f6c57a9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:45:01,692-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [5b637cc9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:45:01,693-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [5b637cc9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:45:01,823-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4b3a84eb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:45:01,823-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4b3a84eb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: b99d405
>2018-05-29 17:45:01,827-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4b3a84eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:45:01,828-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4b3a84eb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:45:01,831-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4b3a84eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:45:01,832-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4b3a84eb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:45:01,836-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4b3a84eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:45:01,837-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4b3a84eb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:45:01,840-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4b3a84eb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:45:01,843-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4b3a84eb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4b3a84eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:45:01,845-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2a718dee
>2018-05-29 17:45:01,845-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:45:01,943-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:45:01,943-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:45:01,943-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:45:01,943-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:45:01,943-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2a718dee
>2018-05-29 17:45:01,943-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:45:01,957-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1e71229f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:45:01,958-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1e71229f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ec10cb6
>2018-05-29 17:45:01,960-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1e71229f] FINISH, SetVdsStatusVDSCommand, log id: ec10cb6
>2018-05-29 17:45:01,995-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1e71229f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1e71229f, Job ID: c12f5ce0-9d48-4aa4-9606-32b7e96eafce, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:45:01,999-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1e71229f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1100b670, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:45:02,001-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1e71229f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1100b670, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:45:02,013-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [250aa785] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:45:02,014-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [250aa785] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2c8de916] Autorecovering 2 hosts
>2018-05-29 17:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2c8de916] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:50:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [bc636f1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:50:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [bc636f1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:50:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [bc636f1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:50:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [bc636f1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:50:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [bc636f1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d027e39
>2018-05-29 17:50:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [bc636f1] FINISH, SetVdsStatusVDSCommand, log id: d027e39
>2018-05-29 17:50:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [bc636f1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:50:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [bc636f1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:50:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [bc636f1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:50:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7cdaac7c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7cdaac7c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7cdaac7c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7cdaac7c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:50:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7cdaac7c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1ae9b7b5
>2018-05-29 17:50:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7cdaac7c] FINISH, SetVdsStatusVDSCommand, log id: 1ae9b7b5
>2018-05-29 17:50:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7cdaac7c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2c38c075
>2018-05-29 17:50:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7cdaac7c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2c38c075
>2018-05-29 17:50:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7cdaac7c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:50:00,100-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7cdaac7c] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 7cdaac7c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-29 17:50:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7cdaac7c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:50:02,316-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [33d96b92] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 23540cdb
>2018-05-29 17:50:02,318-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [33d96b92] FINISH, GetHardwareInfoVDSCommand, log id: 23540cdb
>2018-05-29 17:50:02,331-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3d1511fb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:50:02,339-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [73b2ed16] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:50:02,343-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5ff437d4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:50:02,355-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5ff437d4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1a73f122
>2018-05-29 17:50:02,662-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [29fe6b34] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1ef344a2
>2018-05-29 17:50:02,664-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [29fe6b34] FINISH, GetHardwareInfoVDSCommand, log id: 1ef344a2
>2018-05-29 17:50:02,676-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [608bb065] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:50:02,681-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [43f11178] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:50:02,686-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [e28b368] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:50:02,692-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [e28b368] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6c1c2a6c
>2018-05-29 17:50:02,733-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ff437d4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:50:02,733-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5ff437d4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1a73f122
>2018-05-29 17:50:02,737-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ff437d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:50:02,737-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5ff437d4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:50:02,741-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ff437d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:50:02,742-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5ff437d4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:50:02,747-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ff437d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:50:02,747-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5ff437d4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:50:02,752-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5ff437d4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:50:02,756-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ff437d4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5ff437d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:50:02,759-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: dd6db2a
>2018-05-29 17:50:02,759-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:50:02,858-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:50:02,858-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:50:02,858-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:50:02,858-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:50:02,858-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: dd6db2a
>2018-05-29 17:50:02,858-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:50:02,872-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [31c278ad] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:50:02,873-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [31c278ad] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6472732e
>2018-05-29 17:50:02,875-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [31c278ad] FINISH, SetVdsStatusVDSCommand, log id: 6472732e
>2018-05-29 17:50:02,905-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [31c278ad] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 31c278ad, Job ID: 549fc686-ea46-4573-ad9e-67114a1d353d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:50:02,909-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [31c278ad] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 73b2ed16, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:50:02,912-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [31c278ad] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 73b2ed16, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:50:02,923-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [693520a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:50:02,924-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [693520a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:50:03,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e28b368] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:50:03,064-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [e28b368] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6c1c2a6c
>2018-05-29 17:50:03,067-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e28b368] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:50:03,068-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [e28b368] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:50:03,072-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e28b368] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:50:03,073-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [e28b368] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:50:03,076-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e28b368] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:50:03,077-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [e28b368] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:50:03,080-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [e28b368] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:50:03,082-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e28b368] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: e28b368, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:50:03,085-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7fa9978a
>2018-05-29 17:50:03,085-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:50:03,186-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:50:03,186-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:50:03,186-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:50:03,186-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:50:03,186-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7fa9978a
>2018-05-29 17:50:03,186-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:50:03,200-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [6feb6481] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:50:03,201-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6feb6481] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6a7ec597
>2018-05-29 17:50:03,203-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6feb6481] FINISH, SetVdsStatusVDSCommand, log id: 6a7ec597
>2018-05-29 17:50:03,209-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6feb6481] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6feb6481, Job ID: 41b9ccc6-cd1b-4195-b1a3-4050de7d8d4f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:50:03,214-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6feb6481] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 43f11178, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:50:03,217-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6feb6481] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 43f11178, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:50:03,227-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [fe64888] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:50:03,229-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [fe64888] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:54:43,136-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler4) [2d9ed0ad] Attempting to update VMs/Templates Ovf.
>2018-05-29 17:54:43,139-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 17:54:43,139-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 17:54:43,140-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:54:43,194-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 17:54:43,197-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 17:54:43,197-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 17:54:43,197-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 17:54:43,197-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 17:54:43,198-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 17:54:43,199-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 17:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [60f77b93] Autorecovering 2 hosts
>2018-05-29 17:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [60f77b93] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:55:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [77b99aaf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:55:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [77b99aaf] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:55:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [77b99aaf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:55:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [77b99aaf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:55:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [77b99aaf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26057406
>2018-05-29 17:55:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [77b99aaf] FINISH, SetVdsStatusVDSCommand, log id: 26057406
>2018-05-29 17:55:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [77b99aaf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:55:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [77b99aaf] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 17:55:00,104-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [77b99aaf] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:55:00,110-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5e343b9b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:55:00,116-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5e343b9b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 17:55:00,116-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5e343b9b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:55:00,116-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5e343b9b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:55:00,118-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5e343b9b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 9c999a5
>2018-05-29 17:55:00,120-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5e343b9b] FINISH, SetVdsStatusVDSCommand, log id: 9c999a5
>2018-05-29 17:55:00,123-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5e343b9b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 79d0f0b4
>2018-05-29 17:55:00,125-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5e343b9b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 79d0f0b4
>2018-05-29 17:55:00,125-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5e343b9b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 17:55:00,126-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5e343b9b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 17:55:00,577-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [74b786b8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 23235ffd
>2018-05-29 17:55:00,578-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [74b786b8] FINISH, GetHardwareInfoVDSCommand, log id: 23235ffd
>2018-05-29 17:55:00,591-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [7488fe8c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:55:00,599-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [514e75de] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:55:00,606-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2792c42b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:55:00,663-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2792c42b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 79d51b61
>2018-05-29 17:55:00,871-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3d3a5d5c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 47ea087d
>2018-05-29 17:55:00,873-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3d3a5d5c] FINISH, GetHardwareInfoVDSCommand, log id: 47ea087d
>2018-05-29 17:55:00,886-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [5bbfc60e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:55:00,890-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [319418ec] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:55:00,895-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7061978e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 17:55:00,901-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7061978e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6a3154ec
>2018-05-29 17:55:01,043-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2792c42b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:55:01,043-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2792c42b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 79d51b61
>2018-05-29 17:55:01,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2792c42b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:55:01,048-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2792c42b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:55:01,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2792c42b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:55:01,053-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2792c42b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:55:01,057-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2792c42b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:55:01,058-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2792c42b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:55:01,060-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2792c42b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:55:01,063-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2792c42b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2792c42b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:55:01,069-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 26669b27
>2018-05-29 17:55:01,069-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:55:01,163-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:55:01,163-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:55:01,163-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:55:01,163-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:55:01,163-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 26669b27
>2018-05-29 17:55:01,163-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:55:01,177-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [44384b1b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:55:01,178-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [44384b1b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1ecaa474
>2018-05-29 17:55:01,180-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [44384b1b] FINISH, SetVdsStatusVDSCommand, log id: 1ecaa474
>2018-05-29 17:55:01,188-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44384b1b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 44384b1b, Job ID: ac591d78-3629-4884-831e-4b944e4ff9cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:55:01,193-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44384b1b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 514e75de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:55:01,195-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44384b1b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 514e75de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:55:01,207-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [5d399415] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 17:55:01,208-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [5d399415] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 17:55:02,272-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7061978e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 17:55:02,272-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7061978e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6a3154ec
>2018-05-29 17:55:02,276-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7061978e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:55:02,277-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7061978e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:55:02,281-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7061978e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:55:02,282-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7061978e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:55:02,285-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7061978e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 17:55:02,286-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7061978e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 17:55:02,290-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7061978e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 17:55:02,294-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7061978e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7061978e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 17:55:02,297-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: c3b58b
>2018-05-29 17:55:02,297-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 17:55:02,394-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:55:02,394-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 17:55:02,394-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 17:55:02,394-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 17:55:02,394-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: c3b58b
>2018-05-29 17:55:02,394-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 17:55:02,409-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [3ccb237d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:55:02,410-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3ccb237d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 624f4b8
>2018-05-29 17:55:02,412-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3ccb237d] FINISH, SetVdsStatusVDSCommand, log id: 624f4b8
>2018-05-29 17:55:02,444-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3ccb237d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3ccb237d, Job ID: a1411b9b-a881-4e0d-9273-28dff826d3b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 17:55:02,449-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3ccb237d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 319418ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 17:55:02,451-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3ccb237d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 319418ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 17:55:02,462-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [354728f6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 17:55:02,463-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [354728f6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [79e0ef4a] Autorecovering 2 hosts
>2018-05-29 18:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [79e0ef4a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:00:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [69815e56] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:00:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [69815e56] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:00:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [69815e56] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:00:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [69815e56] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:00:00,080-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [69815e56] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32def646
>2018-05-29 18:00:00,083-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [69815e56] FINISH, SetVdsStatusVDSCommand, log id: 32def646
>2018-05-29 18:00:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [69815e56] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:00:00,089-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [69815e56] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 69815e56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-29 18:00:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [69815e56] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:00:00,090-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [69815e56] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:00:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4fac15d7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:00:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4fac15d7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:00:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4fac15d7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:00:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4fac15d7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:00:00,104-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4fac15d7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 77433c20
>2018-05-29 18:00:00,106-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4fac15d7] FINISH, SetVdsStatusVDSCommand, log id: 77433c20
>2018-05-29 18:00:00,108-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [4fac15d7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 23a6fbcf
>2018-05-29 18:00:00,111-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [4fac15d7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 23a6fbcf
>2018-05-29 18:00:00,111-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4fac15d7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:00:00,112-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4fac15d7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:00:01,963-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [20593be9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1820f201
>2018-05-29 18:00:01,965-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [20593be9] FINISH, GetHardwareInfoVDSCommand, log id: 1820f201
>2018-05-29 18:00:01,978-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [ba9d38c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:00:01,987-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [319a5c07] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:00:01,992-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [138afedc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:00:02,003-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [138afedc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 763ab777
>2018-05-29 18:00:02,384-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [138afedc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:00:02,384-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [138afedc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 763ab777
>2018-05-29 18:00:02,389-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [138afedc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:00:02,389-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [138afedc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:00:02,393-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [138afedc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:00:02,394-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [138afedc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:00:02,398-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [138afedc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:00:02,399-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [138afedc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:00:02,401-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [138afedc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:00:02,404-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [138afedc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 138afedc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:00:02,411-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3c46517e
>2018-05-29 18:00:02,411-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:00:02,510-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:00:02,510-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:00:02,510-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:00:02,510-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:00:02,510-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3c46517e
>2018-05-29 18:00:02,510-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:00:02,525-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [1f48ba20] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:00:02,526-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1f48ba20] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 105772d8
>2018-05-29 18:00:02,528-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1f48ba20] FINISH, SetVdsStatusVDSCommand, log id: 105772d8
>2018-05-29 18:00:02,558-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1f48ba20] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1f48ba20, Job ID: 031eb7df-283f-40a0-8cef-5ce4e5e45767, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:00:02,563-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1f48ba20] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 319a5c07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:00:02,566-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1f48ba20] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 319a5c07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:00:02,577-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [4398f53f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:00:02,579-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [4398f53f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:00:03,214-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [718966cc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 197790b4
>2018-05-29 18:00:03,216-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [718966cc] FINISH, GetHardwareInfoVDSCommand, log id: 197790b4
>2018-05-29 18:00:03,230-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [186cb42b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:00:03,238-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [1e03a96b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:00:03,244-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7ccc71d1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:00:03,253-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7ccc71d1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 56eb91f9
>2018-05-29 18:00:03,629-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7ccc71d1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:00:03,629-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7ccc71d1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 56eb91f9
>2018-05-29 18:00:03,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7ccc71d1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:00:03,634-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7ccc71d1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:00:03,638-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7ccc71d1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:00:03,638-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7ccc71d1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:00:03,642-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7ccc71d1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:00:03,643-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7ccc71d1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:00:03,647-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7ccc71d1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:00:03,650-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7ccc71d1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7ccc71d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:00:03,654-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5eb61d93
>2018-05-29 18:00:03,654-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:00:03,753-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:00:03,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:00:03,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:00:03,753-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:00:03,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5eb61d93
>2018-05-29 18:00:03,753-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:00:03,769-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [5d7b44a5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:00:03,770-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5d7b44a5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: c78eaa8
>2018-05-29 18:00:03,773-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5d7b44a5] FINISH, SetVdsStatusVDSCommand, log id: c78eaa8
>2018-05-29 18:00:03,808-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d7b44a5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5d7b44a5, Job ID: 9096be2d-e080-40e8-90c5-397c52773c2a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:00:03,813-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d7b44a5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1e03a96b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:00:03,815-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d7b44a5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1e03a96b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:00:03,828-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6a227fe2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:00:03,829-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6a227fe2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3b9555b7] Autorecovering 2 hosts
>2018-05-29 18:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3b9555b7] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:05:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5f9d1284] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:05:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5f9d1284] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:05:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5f9d1284] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:05:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5f9d1284] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:05:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5f9d1284] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5204f7a9
>2018-05-29 18:05:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5f9d1284] FINISH, SetVdsStatusVDSCommand, log id: 5204f7a9
>2018-05-29 18:05:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5f9d1284] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:05:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5f9d1284] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:05:00,072-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5f9d1284] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:05:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7424ce9a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:05:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7424ce9a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:05:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7424ce9a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:05:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7424ce9a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:05:00,086-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7424ce9a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d5ed2d4
>2018-05-29 18:05:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7424ce9a] FINISH, SetVdsStatusVDSCommand, log id: d5ed2d4
>2018-05-29 18:05:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [7424ce9a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 647a5e8f
>2018-05-29 18:05:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [7424ce9a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 647a5e8f
>2018-05-29 18:05:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7424ce9a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:05:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7424ce9a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:05:01,508-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [502e1f91] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2f82ecba
>2018-05-29 18:05:01,510-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [502e1f91] FINISH, GetHardwareInfoVDSCommand, log id: 2f82ecba
>2018-05-29 18:05:01,523-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [328ad08e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:05:01,528-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [5083a6ed] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:05:01,533-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [287eae44] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:05:01,540-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [287eae44] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7861072d
>2018-05-29 18:05:01,902-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [287eae44] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:05:01,902-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [287eae44] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7861072d
>2018-05-29 18:05:01,905-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [287eae44] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:05:01,906-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [287eae44] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:05:01,910-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [287eae44] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:05:01,911-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [287eae44] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:05:01,915-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [287eae44] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:05:01,915-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [287eae44] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:05:01,918-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [287eae44] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:05:01,921-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [287eae44] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 287eae44, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:05:01,923-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3fcea265
>2018-05-29 18:05:01,923-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:05:02,020-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:05:02,020-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:05:02,020-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:05:02,020-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:05:02,020-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3fcea265
>2018-05-29 18:05:02,020-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:05:02,034-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1b315668] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:05:02,035-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1b315668] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b625226
>2018-05-29 18:05:02,037-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1b315668] FINISH, SetVdsStatusVDSCommand, log id: 1b625226
>2018-05-29 18:05:02,043-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1b315668] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1b315668, Job ID: b79e59d2-4585-4e40-aced-df9f141fbbe7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:05:02,048-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1b315668] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5083a6ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:05:02,050-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1b315668] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5083a6ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:05:02,061-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [4f86a98] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:05:02,062-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [4f86a98] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:05:03,201-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [10866b04] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1eb1c87b
>2018-05-29 18:05:03,203-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [10866b04] FINISH, GetHardwareInfoVDSCommand, log id: 1eb1c87b
>2018-05-29 18:05:03,215-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [7f403594] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:05:03,220-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [7b010592] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:05:03,225-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [76b9046b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:05:03,232-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [76b9046b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 739849a1
>2018-05-29 18:05:03,615-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [76b9046b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:05:03,615-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [76b9046b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 739849a1
>2018-05-29 18:05:03,619-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [76b9046b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:05:03,620-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [76b9046b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:05:03,624-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [76b9046b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:05:03,625-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [76b9046b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:05:03,629-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [76b9046b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:05:03,630-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [76b9046b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:05:03,633-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [76b9046b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:05:03,636-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [76b9046b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 76b9046b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:05:03,639-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5cf78534
>2018-05-29 18:05:03,639-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:05:03,738-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:05:03,738-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:05:03,738-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:05:03,738-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:05:03,738-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5cf78534
>2018-05-29 18:05:03,738-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:05:03,751-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [b98bfb1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:05:03,753-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [b98bfb1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18afd2b5
>2018-05-29 18:05:03,755-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [b98bfb1] FINISH, SetVdsStatusVDSCommand, log id: 18afd2b5
>2018-05-29 18:05:03,761-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [b98bfb1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: b98bfb1, Job ID: a357ffcc-04bc-400f-b85b-76a4535d9ec4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:05:03,766-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [b98bfb1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7b010592, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:05:03,768-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [b98bfb1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7b010592, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:05:03,780-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [7706ca68] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:05:03,781-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7706ca68] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3d0b0a10] Autorecovering 2 hosts
>2018-05-29 18:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3d0b0a10] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:10:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7cf654f7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:10:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7cf654f7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:10:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7cf654f7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:10:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7cf654f7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:10:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7cf654f7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6981f238
>2018-05-29 18:10:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7cf654f7] FINISH, SetVdsStatusVDSCommand, log id: 6981f238
>2018-05-29 18:10:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7cf654f7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:10:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7cf654f7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:10:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7cf654f7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:10:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [528f3189] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:10:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [528f3189] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:10:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [528f3189] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:10:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [528f3189] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:10:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [528f3189] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 60143544
>2018-05-29 18:10:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [528f3189] FINISH, SetVdsStatusVDSCommand, log id: 60143544
>2018-05-29 18:10:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [528f3189] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: b4f5b82
>2018-05-29 18:10:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [528f3189] FINISH, SetHaMaintenanceModeVDSCommand, log id: b4f5b82
>2018-05-29 18:10:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [528f3189] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:10:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [528f3189] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:10:01,536-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [30583684] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 632bce3a
>2018-05-29 18:10:01,537-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [30583684] FINISH, GetHardwareInfoVDSCommand, log id: 632bce3a
>2018-05-29 18:10:01,552-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [258ec9c3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:10:01,560-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [4946a533] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:10:01,566-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [52ba9ad8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:10:01,577-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [52ba9ad8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4c632207
>2018-05-29 18:10:01,956-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [52ba9ad8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:10:01,956-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [52ba9ad8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4c632207
>2018-05-29 18:10:01,960-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [52ba9ad8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:10:01,961-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [52ba9ad8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:10:01,965-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [52ba9ad8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:10:01,965-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [52ba9ad8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:10:01,969-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [52ba9ad8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:10:01,970-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [52ba9ad8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:10:01,973-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [52ba9ad8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:10:01,975-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [52ba9ad8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 52ba9ad8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:10:01,981-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 75e40713
>2018-05-29 18:10:01,982-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:10:02,079-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:10:02,079-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:10:02,079-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:10:02,079-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:10:02,079-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 75e40713
>2018-05-29 18:10:02,079-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:10:02,092-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [20de6c01] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:10:02,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [20de6c01] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19029b5a
>2018-05-29 18:10:02,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [20de6c01] FINISH, SetVdsStatusVDSCommand, log id: 19029b5a
>2018-05-29 18:10:02,127-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20de6c01] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 20de6c01, Job ID: fc458559-f2ab-4b28-a865-3a19c63c433b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:10:02,132-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20de6c01] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4946a533, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:10:02,134-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20de6c01] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4946a533, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:10:02,145-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [40fdaaaf] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:10:02,146-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [40fdaaaf] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:10:02,760-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [454ae2b8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7186aa0d
>2018-05-29 18:10:02,762-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [454ae2b8] FINISH, GetHardwareInfoVDSCommand, log id: 7186aa0d
>2018-05-29 18:10:02,775-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [307ea221] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:10:02,779-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [39317f9b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:10:02,785-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5cf76ec0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:10:02,791-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5cf76ec0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 473d80e7
>2018-05-29 18:10:03,160-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5cf76ec0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:10:03,160-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5cf76ec0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 473d80e7
>2018-05-29 18:10:03,164-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5cf76ec0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:10:03,165-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5cf76ec0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:10:03,169-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5cf76ec0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:10:03,169-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5cf76ec0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:10:03,173-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5cf76ec0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:10:03,174-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5cf76ec0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:10:03,177-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5cf76ec0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:10:03,180-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5cf76ec0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5cf76ec0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:10:03,183-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3857c5d3
>2018-05-29 18:10:03,183-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:10:03,281-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:10:03,281-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:10:03,281-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:10:03,281-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:10:03,281-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3857c5d3
>2018-05-29 18:10:03,281-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:10:03,294-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [72114d4f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:10:03,295-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [72114d4f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3275f4de
>2018-05-29 18:10:03,297-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [72114d4f] FINISH, SetVdsStatusVDSCommand, log id: 3275f4de
>2018-05-29 18:10:03,303-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [72114d4f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 72114d4f, Job ID: ca215b4b-b84c-44c5-9ef5-5d49fc885971, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:10:03,308-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [72114d4f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 39317f9b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:10:03,310-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [72114d4f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 39317f9b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:10:03,320-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [673958c2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:10:03,321-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [673958c2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1023ff79] Autorecovering 2 hosts
>2018-05-29 18:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1023ff79] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:15:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ad83f66] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ad83f66] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ad83f66] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ad83f66] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:15:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3ad83f66] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 55da19a9
>2018-05-29 18:15:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3ad83f66] FINISH, SetVdsStatusVDSCommand, log id: 55da19a9
>2018-05-29 18:15:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ad83f66] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:15:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ad83f66] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:15:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3ad83f66] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:15:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2cb13f87] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:15:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2cb13f87] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:15:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2cb13f87] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:15:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2cb13f87] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:15:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2cb13f87] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 74b035e1
>2018-05-29 18:15:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2cb13f87] FINISH, SetVdsStatusVDSCommand, log id: 74b035e1
>2018-05-29 18:15:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2cb13f87] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 63754fad
>2018-05-29 18:15:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2cb13f87] FINISH, SetHaMaintenanceModeVDSCommand, log id: 63754fad
>2018-05-29 18:15:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2cb13f87] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:15:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2cb13f87] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:15:00,985-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [39e18ff8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5b89b23c
>2018-05-29 18:15:00,987-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [39e18ff8] FINISH, GetHardwareInfoVDSCommand, log id: 5b89b23c
>2018-05-29 18:15:01,000-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [4b4cb38c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:15:01,009-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [616800e8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:15:01,014-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [536ee703] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:15:01,021-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [536ee703] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23081148
>2018-05-29 18:15:02,387-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [536ee703] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:15:02,387-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [536ee703] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 23081148
>2018-05-29 18:15:02,391-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [536ee703] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:15:02,392-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [536ee703] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:15:02,396-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [536ee703] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:15:02,397-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [536ee703] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:15:02,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [536ee703] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:15:02,402-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [536ee703] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:15:02,405-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [536ee703] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:15:02,408-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [536ee703] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 536ee703, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:15:02,411-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 253eb1d2
>2018-05-29 18:15:02,411-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:15:02,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:15:02,508-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:15:02,508-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:15:02,508-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:15:02,508-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 253eb1d2
>2018-05-29 18:15:02,508-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:15:02,522-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [4119140c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:15:02,523-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4119140c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 10e600b3
>2018-05-29 18:15:02,526-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4119140c] FINISH, SetVdsStatusVDSCommand, log id: 10e600b3
>2018-05-29 18:15:02,556-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4119140c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4119140c, Job ID: a684a8b5-b6e8-4568-b337-e0d5c6d1d8fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:15:02,560-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4119140c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 616800e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:15:02,563-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4119140c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 616800e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:15:02,573-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [35f216a2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:15:02,574-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [35f216a2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:15:03,105-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3139ab6c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 38a94cc2
>2018-05-29 18:15:03,107-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3139ab6c] FINISH, GetHardwareInfoVDSCommand, log id: 38a94cc2
>2018-05-29 18:15:03,121-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [71ece68] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:15:03,125-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [41964c19] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:15:03,130-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [397c1340] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:15:03,136-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [397c1340] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4629bb8b
>2018-05-29 18:15:03,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [397c1340] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:15:03,517-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [397c1340] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4629bb8b
>2018-05-29 18:15:03,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [397c1340] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:15:03,522-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [397c1340] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:15:03,526-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [397c1340] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:15:03,526-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [397c1340] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:15:03,530-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [397c1340] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:15:03,531-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [397c1340] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:15:03,533-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [397c1340] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:15:03,536-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [397c1340] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 397c1340, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:15:03,539-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 53848e24
>2018-05-29 18:15:03,539-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:15:03,637-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:15:03,637-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:15:03,637-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:15:03,637-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:15:03,637-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 53848e24
>2018-05-29 18:15:03,637-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:15:03,650-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [2d51b87d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:15:03,651-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2d51b87d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43b28d56
>2018-05-29 18:15:03,653-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2d51b87d] FINISH, SetVdsStatusVDSCommand, log id: 43b28d56
>2018-05-29 18:15:03,659-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2d51b87d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2d51b87d, Job ID: d75c5058-cd28-41d7-8d5d-2af2915ab3b8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:15:03,664-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2d51b87d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 41964c19, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:15:03,666-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2d51b87d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 41964c19, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:15:03,677-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [c0cc647] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:15:03,678-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [c0cc647] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:20:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5074d680] Autorecovering 2 hosts
>2018-05-29 18:20:00,011-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5074d680] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ca2b874] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:20:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ca2b874] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:20:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ca2b874] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:20:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ca2b874] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:20:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5ca2b874] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 60ac05bc
>2018-05-29 18:20:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5ca2b874] FINISH, SetVdsStatusVDSCommand, log id: 60ac05bc
>2018-05-29 18:20:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ca2b874] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:20:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ca2b874] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:20:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5ca2b874] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:20:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [713ae1ac] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:20:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [713ae1ac] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:20:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [713ae1ac] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:20:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [713ae1ac] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:20:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [713ae1ac] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 152bb54c
>2018-05-29 18:20:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [713ae1ac] FINISH, SetVdsStatusVDSCommand, log id: 152bb54c
>2018-05-29 18:20:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [713ae1ac] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 391791d9
>2018-05-29 18:20:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [713ae1ac] FINISH, SetHaMaintenanceModeVDSCommand, log id: 391791d9
>2018-05-29 18:20:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [713ae1ac] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:20:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [713ae1ac] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:20:00,343-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [27c4da09] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7ae44862
>2018-05-29 18:20:00,345-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [27c4da09] FINISH, GetHardwareInfoVDSCommand, log id: 7ae44862
>2018-05-29 18:20:00,358-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [64772df] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:20:00,363-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [60544686] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:20:00,369-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [344ce3b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:20:00,381-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [344ce3b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c1d136f
>2018-05-29 18:20:00,754-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [344ce3b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:20:00,754-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [344ce3b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c1d136f
>2018-05-29 18:20:00,758-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [344ce3b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:20:00,759-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [344ce3b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:20:00,762-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [344ce3b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:20:00,763-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [344ce3b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:20:00,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [344ce3b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:20:00,768-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [344ce3b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:20:00,770-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [344ce3b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:20:00,773-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [344ce3b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 344ce3b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:20:00,780-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 36911c36
>2018-05-29 18:20:00,780-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:20:00,877-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:20:00,877-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:20:00,877-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:20:00,877-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:20:00,877-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 36911c36
>2018-05-29 18:20:00,877-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:20:00,890-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [16aa4af3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:20:00,891-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [16aa4af3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3cc4953b
>2018-05-29 18:20:00,894-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [16aa4af3] FINISH, SetVdsStatusVDSCommand, log id: 3cc4953b
>2018-05-29 18:20:00,902-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [16aa4af3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 16aa4af3, Job ID: 6238b244-6aec-4cac-8f34-085bb1e5e386, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:20:00,907-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [16aa4af3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 60544686, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:20:00,909-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [16aa4af3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 60544686, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:20:00,920-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [364bae94] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:20:00,921-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [364bae94] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:20:01,283-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [37539e8a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 323ba2f8
>2018-05-29 18:20:01,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [37539e8a] FINISH, GetHardwareInfoVDSCommand, log id: 323ba2f8
>2018-05-29 18:20:01,297-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [587cfc0d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:20:01,302-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [52b89977] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:20:01,307-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [4f14598d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:20:01,314-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [4f14598d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 64d64a9b
>2018-05-29 18:20:01,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f14598d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:20:01,690-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [4f14598d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 64d64a9b
>2018-05-29 18:20:01,694-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f14598d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:20:01,694-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [4f14598d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:20:01,698-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f14598d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:20:01,699-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [4f14598d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:20:01,703-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f14598d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:20:01,704-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [4f14598d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:20:01,706-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [4f14598d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:20:01,709-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f14598d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4f14598d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:20:01,711-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7b237949
>2018-05-29 18:20:01,711-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:20:01,810-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:20:01,810-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:20:01,810-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:20:01,810-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:20:01,810-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7b237949
>2018-05-29 18:20:01,810-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:20:01,823-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [56a30e48] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:20:01,824-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [56a30e48] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 27237d2b
>2018-05-29 18:20:01,826-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [56a30e48] FINISH, SetVdsStatusVDSCommand, log id: 27237d2b
>2018-05-29 18:20:01,831-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [56a30e48] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 56a30e48, Job ID: c601316f-6872-4ffd-a5a1-e095e4f40a99, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:20:01,835-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [56a30e48] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 52b89977, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:20:01,837-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [56a30e48] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 52b89977, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:20:01,848-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [6c21268e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:20:01,849-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [6c21268e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [79a15e26] Autorecovering 2 hosts
>2018-05-29 18:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [79a15e26] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:25:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [125e5cac] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [125e5cac] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [125e5cac] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [125e5cac] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:25:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [125e5cac] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 82190f7
>2018-05-29 18:25:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [125e5cac] FINISH, SetVdsStatusVDSCommand, log id: 82190f7
>2018-05-29 18:25:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [125e5cac] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:25:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [125e5cac] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:25:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [125e5cac] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:25:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1804dcab] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1804dcab] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1804dcab] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1804dcab] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:25:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1804dcab] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 28b616d3
>2018-05-29 18:25:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1804dcab] FINISH, SetVdsStatusVDSCommand, log id: 28b616d3
>2018-05-29 18:25:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [1804dcab] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 293f436e
>2018-05-29 18:25:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [1804dcab] FINISH, SetHaMaintenanceModeVDSCommand, log id: 293f436e
>2018-05-29 18:25:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1804dcab] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:25:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1804dcab] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:25:01,651-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2366417a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 130b52c3
>2018-05-29 18:25:01,653-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2366417a] FINISH, GetHardwareInfoVDSCommand, log id: 130b52c3
>2018-05-29 18:25:01,666-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6fb87457] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:25:01,675-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [17dcf698] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:25:01,680-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [369f0dbf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:25:01,691-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [369f0dbf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f73013b
>2018-05-29 18:25:02,066-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [369f0dbf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:25:02,066-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [369f0dbf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f73013b
>2018-05-29 18:25:02,071-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [369f0dbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:25:02,071-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [369f0dbf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:25:02,075-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [369f0dbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:25:02,076-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [369f0dbf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:25:02,080-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [369f0dbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:25:02,081-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [369f0dbf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:25:02,083-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [369f0dbf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:25:02,086-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [369f0dbf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 369f0dbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:25:02,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2074a25a
>2018-05-29 18:25:02,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:25:02,195-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:25:02,195-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:25:02,195-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:25:02,195-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:25:02,195-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2074a25a
>2018-05-29 18:25:02,195-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:25:02,208-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [8522658] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:25:02,209-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [8522658] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 38111abe
>2018-05-29 18:25:02,211-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [8522658] FINISH, SetVdsStatusVDSCommand, log id: 38111abe
>2018-05-29 18:25:02,244-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8522658] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 8522658, Job ID: 1372909b-f997-4282-85ce-8ce1ebd61a0f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:25:02,248-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8522658] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17dcf698, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:25:02,251-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8522658] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17dcf698, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:25:02,261-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [59e52237] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:25:02,262-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [59e52237] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:25:02,505-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [7eb5975f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 389f16fe
>2018-05-29 18:25:02,506-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [7eb5975f] FINISH, GetHardwareInfoVDSCommand, log id: 389f16fe
>2018-05-29 18:25:02,519-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6e03177c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:25:02,552-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [7f72dfd3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:25:02,557-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [593dee26] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:25:02,564-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [593dee26] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2f4f9a08
>2018-05-29 18:25:02,939-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [593dee26] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:25:02,939-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [593dee26] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2f4f9a08
>2018-05-29 18:25:02,943-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [593dee26] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:25:02,944-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [593dee26] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:25:02,948-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [593dee26] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:25:02,949-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [593dee26] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:25:02,953-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [593dee26] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:25:02,953-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [593dee26] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:25:02,957-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [593dee26] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:25:02,959-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [593dee26] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 593dee26, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:25:02,962-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: ef8c042
>2018-05-29 18:25:02,962-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:25:03,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:25:03,056-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:25:03,056-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:25:03,056-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:25:03,056-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: ef8c042
>2018-05-29 18:25:03,056-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:25:03,069-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [34775689] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:25:03,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [34775689] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 20d3988f
>2018-05-29 18:25:03,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [34775689] FINISH, SetVdsStatusVDSCommand, log id: 20d3988f
>2018-05-29 18:25:03,078-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [34775689] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 34775689, Job ID: cf6b40da-cd48-46c7-b012-43f5f56582c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:25:03,082-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [34775689] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7f72dfd3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:25:03,084-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [34775689] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7f72dfd3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:25:03,095-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [6850579b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:25:03,096-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [6850579b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6d93462] Autorecovering 2 hosts
>2018-05-29 18:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6d93462] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:30:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7f529150] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7f529150] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7f529150] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7f529150] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:30:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7f529150] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 749ef548
>2018-05-29 18:30:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7f529150] FINISH, SetVdsStatusVDSCommand, log id: 749ef548
>2018-05-29 18:30:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7f529150] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:30:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7f529150] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:30:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7f529150] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:30:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [589e2afc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:30:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [589e2afc] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:30:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [589e2afc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:30:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [589e2afc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:30:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [589e2afc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3892098c
>2018-05-29 18:30:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [589e2afc] FINISH, SetVdsStatusVDSCommand, log id: 3892098c
>2018-05-29 18:30:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [589e2afc] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6d7d20ed
>2018-05-29 18:30:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [589e2afc] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6d7d20ed
>2018-05-29 18:30:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [589e2afc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:30:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [589e2afc] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:30:00,697-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [b1bc22e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 33a61ac9
>2018-05-29 18:30:00,699-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [b1bc22e] FINISH, GetHardwareInfoVDSCommand, log id: 33a61ac9
>2018-05-29 18:30:00,712-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [20f766f8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:30:00,720-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [315f33d9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:30:00,726-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [663e8ea9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:30:00,737-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [663e8ea9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 65e3ecc
>2018-05-29 18:30:01,117-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [663e8ea9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:30:01,117-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [663e8ea9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 65e3ecc
>2018-05-29 18:30:01,121-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [663e8ea9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:30:01,122-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [663e8ea9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:30:01,125-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [663e8ea9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:30:01,126-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [663e8ea9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:30:01,130-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [663e8ea9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:30:01,130-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [663e8ea9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:30:01,133-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [663e8ea9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:30:01,136-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [663e8ea9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 663e8ea9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:30:01,142-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 44e5f48a
>2018-05-29 18:30:01,142-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:30:01,240-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-11) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:30:01,240-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:30:01,240-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:30:01,240-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:30:01,240-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] FINISH, ConnectStoragePoolVDSCommand, log id: 44e5f48a
>2018-05-29 18:30:01,240-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-11) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:30:01,253-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [1d6f39db] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:30:01,254-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1d6f39db] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6051c7fc
>2018-05-29 18:30:01,256-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1d6f39db] FINISH, SetVdsStatusVDSCommand, log id: 6051c7fc
>2018-05-29 18:30:01,287-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1d6f39db] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1d6f39db, Job ID: 97024c13-3cf1-44d9-a11e-d42d80d68577, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:30:01,291-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1d6f39db] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 315f33d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:30:01,293-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1d6f39db] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 315f33d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:30:01,303-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [28301316] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:30:01,304-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [28301316] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:30:02,968-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1c100426] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 559bc929
>2018-05-29 18:30:02,970-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1c100426] FINISH, GetHardwareInfoVDSCommand, log id: 559bc929
>2018-05-29 18:30:02,982-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [37ef17f7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:30:02,987-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [36c8bca0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:30:02,992-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [12a5c75a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:30:02,999-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [12a5c75a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 216c1ae7
>2018-05-29 18:30:03,366-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [12a5c75a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:30:03,366-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [12a5c75a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 216c1ae7
>2018-05-29 18:30:03,369-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [12a5c75a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:30:03,370-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [12a5c75a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:30:03,374-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [12a5c75a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:30:03,374-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [12a5c75a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:30:03,378-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [12a5c75a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:30:03,379-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [12a5c75a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:30:03,381-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [12a5c75a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:30:03,384-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [12a5c75a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 12a5c75a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:30:03,387-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7fb99575
>2018-05-29 18:30:03,387-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:30:03,484-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:30:03,484-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:30:03,484-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:30:03,484-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:30:03,484-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7fb99575
>2018-05-29 18:30:03,484-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:30:03,497-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [6637cf84] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:30:03,498-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6637cf84] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1319d341
>2018-05-29 18:30:03,500-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6637cf84] FINISH, SetVdsStatusVDSCommand, log id: 1319d341
>2018-05-29 18:30:03,505-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6637cf84] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6637cf84, Job ID: 586aea81-579f-45c6-8d67-910ae4f20a1a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:30:03,510-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6637cf84] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 36c8bca0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:30:03,512-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6637cf84] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 36c8bca0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:30:03,522-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [7af1a33b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:30:03,523-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [7af1a33b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:35:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [725abf5] Autorecovering 2 hosts
>2018-05-29 18:35:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [725abf5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:35:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [766f35f0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:35:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [766f35f0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:35:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [766f35f0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:35:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [766f35f0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:35:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [766f35f0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 337b5125
>2018-05-29 18:35:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [766f35f0] FINISH, SetVdsStatusVDSCommand, log id: 337b5125
>2018-05-29 18:35:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [766f35f0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:35:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [766f35f0] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:35:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [766f35f0] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:35:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5de07a08] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:35:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5de07a08] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:35:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5de07a08] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:35:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5de07a08] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:35:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5de07a08] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 364622fa
>2018-05-29 18:35:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5de07a08] FINISH, SetVdsStatusVDSCommand, log id: 364622fa
>2018-05-29 18:35:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5de07a08] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5247dc61
>2018-05-29 18:35:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5de07a08] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5247dc61
>2018-05-29 18:35:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5de07a08] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:35:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5de07a08] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:35:01,171-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [632ea97f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 20950321
>2018-05-29 18:35:01,173-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [632ea97f] FINISH, GetHardwareInfoVDSCommand, log id: 20950321
>2018-05-29 18:35:01,185-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [64b89242] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:35:01,190-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5fcf7c4b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:35:01,195-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [309ca44e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:35:01,207-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [309ca44e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4731052d
>2018-05-29 18:35:02,020-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1fd7cedb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 18cb7c08
>2018-05-29 18:35:02,022-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1fd7cedb] FINISH, GetHardwareInfoVDSCommand, log id: 18cb7c08
>2018-05-29 18:35:02,034-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [578e31f3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:35:02,039-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [7939dec4] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:35:02,044-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [55dc5b23] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:35:02,050-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [55dc5b23] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6c3a8b68
>2018-05-29 18:35:02,433-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [55dc5b23] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:35:02,433-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [55dc5b23] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6c3a8b68
>2018-05-29 18:35:02,437-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [55dc5b23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:35:02,437-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [55dc5b23] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:35:02,441-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [55dc5b23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:35:02,442-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [55dc5b23] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:35:02,446-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [55dc5b23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:35:02,447-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [55dc5b23] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:35:02,449-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [55dc5b23] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:35:02,452-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [55dc5b23] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 55dc5b23, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:35:02,458-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3bf419cd
>2018-05-29 18:35:02,458-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:35:02,556-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:35:02,556-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:35:02,556-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:35:02,557-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:35:02,557-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3bf419cd
>2018-05-29 18:35:02,557-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:35:02,570-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [62bb8445] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:35:02,572-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [62bb8445] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 656294f8
>2018-05-29 18:35:02,574-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [62bb8445] FINISH, SetVdsStatusVDSCommand, log id: 656294f8
>2018-05-29 18:35:02,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [309ca44e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:35:02,575-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [309ca44e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4731052d
>2018-05-29 18:35:02,579-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [309ca44e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:35:02,580-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [309ca44e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:35:02,584-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [309ca44e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:35:02,584-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [309ca44e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:35:02,588-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [309ca44e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:35:02,589-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [309ca44e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:35:02,607-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [62bb8445] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 62bb8445, Job ID: 6a057ab9-cab3-43f2-9182-5d24c37e1786, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:35:02,613-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [62bb8445] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7939dec4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:35:02,615-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [62bb8445] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7939dec4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:35:02,627-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [35b2c9ba] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:35:02,628-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [35b2c9ba] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:35:02,639-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [309ca44e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:35:02,642-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [309ca44e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 309ca44e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:35:02,649-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5b2fccd5
>2018-05-29 18:35:02,649-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:35:02,745-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:35:02,745-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:35:02,745-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:35:02,745-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:35:02,745-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5b2fccd5
>2018-05-29 18:35:02,745-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:35:02,758-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [c847b4a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:35:02,760-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [c847b4a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45aea58a
>2018-05-29 18:35:02,762-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [c847b4a] FINISH, SetVdsStatusVDSCommand, log id: 45aea58a
>2018-05-29 18:35:02,770-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c847b4a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: c847b4a, Job ID: a100bf56-739a-4aec-9f46-293ca1cfa504, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:35:02,775-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c847b4a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5fcf7c4b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:35:02,777-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c847b4a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5fcf7c4b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:35:02,787-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [7cf4e3dd] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:35:02,789-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7cf4e3dd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:40:00,015-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [4607c14d] Autorecovering 2 hosts
>2018-05-29 18:40:00,015-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [4607c14d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [23441e5c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:40:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [23441e5c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:40:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [23441e5c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:40:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [23441e5c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:40:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [23441e5c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3c98e7af
>2018-05-29 18:40:00,080-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [23441e5c] FINISH, SetVdsStatusVDSCommand, log id: 3c98e7af
>2018-05-29 18:40:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [23441e5c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:40:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [23441e5c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:40:00,084-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [23441e5c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [53dc5c1f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:40:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [53dc5c1f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:40:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [53dc5c1f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:40:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [53dc5c1f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:40:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [53dc5c1f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 9a2b9f
>2018-05-29 18:40:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [53dc5c1f] FINISH, SetVdsStatusVDSCommand, log id: 9a2b9f
>2018-05-29 18:40:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [53dc5c1f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 51c5fad4
>2018-05-29 18:40:00,104-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [53dc5c1f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 51c5fad4
>2018-05-29 18:40:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [53dc5c1f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:40:00,105-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [53dc5c1f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:40:00,253-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [f3fd598] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1edcdcee
>2018-05-29 18:40:00,254-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [f3fd598] FINISH, GetHardwareInfoVDSCommand, log id: 1edcdcee
>2018-05-29 18:40:00,267-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4cdadcc2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:40:00,276-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [229412d8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:40:00,281-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4f071070] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:40:00,342-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4f071070] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1df393c1
>2018-05-29 18:40:00,425-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3796200e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 122e6d6a
>2018-05-29 18:40:00,426-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3796200e] FINISH, GetHardwareInfoVDSCommand, log id: 122e6d6a
>2018-05-29 18:40:00,440-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [989b06a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:40:00,444-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [242870a1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:40:00,449-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [341e0b13] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:40:00,455-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [341e0b13] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: de1c74
>2018-05-29 18:40:00,724-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f071070] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:40:00,724-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4f071070] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1df393c1
>2018-05-29 18:40:00,728-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f071070] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:40:00,728-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4f071070] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:40:00,732-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f071070] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:40:00,733-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4f071070] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:40:00,737-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f071070] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:40:00,738-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4f071070] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:40:00,740-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4f071070] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:40:00,743-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f071070] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4f071070, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:40:00,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5c816e2b
>2018-05-29 18:40:00,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:40:00,819-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [341e0b13] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:40:00,819-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [341e0b13] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: de1c74
>2018-05-29 18:40:00,823-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [341e0b13] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:40:00,824-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [341e0b13] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:40:00,827-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [341e0b13] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:40:00,828-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [341e0b13] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:40:00,832-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [341e0b13] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:40:00,832-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [341e0b13] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:40:00,835-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [341e0b13] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:40:00,837-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [341e0b13] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 341e0b13, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:40:00,845-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:40:00,845-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:40:00,845-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:40:00,845-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:40:00,845-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5c816e2b
>2018-05-29 18:40:00,845-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:40:00,846-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7ecb9a4e
>2018-05-29 18:40:00,846-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:40:00,858-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [7fdc33e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:40:00,859-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7fdc33e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7ab2fdbb
>2018-05-29 18:40:00,861-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7fdc33e] FINISH, SetVdsStatusVDSCommand, log id: 7ab2fdbb
>2018-05-29 18:40:00,893-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fdc33e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7fdc33e, Job ID: 454736f6-27c7-49c5-84e0-f86f80d8304d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:40:00,898-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fdc33e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 229412d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:40:00,900-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fdc33e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 229412d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:40:00,911-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [38d45046] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:40:00,913-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [38d45046] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:40:00,946-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:40:00,947-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:40:00,947-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:40:00,947-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:40:00,947-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7ecb9a4e
>2018-05-29 18:40:00,947-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:40:00,959-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [216b55f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:40:00,960-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [216b55f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6e3ebbf1
>2018-05-29 18:40:00,962-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [216b55f] FINISH, SetVdsStatusVDSCommand, log id: 6e3ebbf1
>2018-05-29 18:40:00,968-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [216b55f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 216b55f, Job ID: a3281513-6e09-4ed5-b248-0801b22557ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:40:00,972-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [216b55f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 242870a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:40:00,974-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [216b55f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 242870a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:40:00,985-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5fc30682] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:40:00,986-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5fc30682] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:45:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [77ac09b0] Autorecovering 2 hosts
>2018-05-29 18:45:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [77ac09b0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:45:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [42d466e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [42d466e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [42d466e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [42d466e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:45:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [42d466e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 318da4a6
>2018-05-29 18:45:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [42d466e] FINISH, SetVdsStatusVDSCommand, log id: 318da4a6
>2018-05-29 18:45:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [42d466e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:45:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [42d466e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:45:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [42d466e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:45:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [491a10af] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [491a10af] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [491a10af] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [491a10af] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:45:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [491a10af] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30723bfc
>2018-05-29 18:45:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [491a10af] FINISH, SetVdsStatusVDSCommand, log id: 30723bfc
>2018-05-29 18:45:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [491a10af] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 756177bf
>2018-05-29 18:45:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [491a10af] FINISH, SetHaMaintenanceModeVDSCommand, log id: 756177bf
>2018-05-29 18:45:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [491a10af] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:45:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [491a10af] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:45:01,512-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2e0418f5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3e2be2b1
>2018-05-29 18:45:01,514-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2e0418f5] FINISH, GetHardwareInfoVDSCommand, log id: 3e2be2b1
>2018-05-29 18:45:01,527-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [2a028eae] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:45:01,535-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [4cf6b3a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:45:01,540-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3bf271af] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:45:01,551-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3bf271af] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 59ba247a
>2018-05-29 18:45:01,611-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [57a3dbf2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7e75d441
>2018-05-29 18:45:01,613-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [57a3dbf2] FINISH, GetHardwareInfoVDSCommand, log id: 7e75d441
>2018-05-29 18:45:01,625-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [6291b814] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:45:01,630-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [3925c7f3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:45:01,635-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [ca58e03] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:45:01,640-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [ca58e03] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5507934a
>2018-05-29 18:45:01,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3bf271af] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:45:01,935-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3bf271af] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 59ba247a
>2018-05-29 18:45:01,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3bf271af] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:45:01,939-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3bf271af] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:45:01,943-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3bf271af] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:45:01,944-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3bf271af] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:45:01,947-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3bf271af] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:45:01,948-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3bf271af] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:45:01,951-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3bf271af] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:45:01,953-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3bf271af] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3bf271af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:45:01,960-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2fafae18
>2018-05-29 18:45:01,960-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:45:02,057-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:45:02,057-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:45:02,057-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:45:02,057-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:45:02,057-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2fafae18
>2018-05-29 18:45:02,057-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:45:02,070-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [1655bb90] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:45:02,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1655bb90] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1323d7ef
>2018-05-29 18:45:02,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1655bb90] FINISH, SetVdsStatusVDSCommand, log id: 1323d7ef
>2018-05-29 18:45:02,105-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1655bb90] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1655bb90, Job ID: 39deaf46-4139-4080-9cfb-817326a77930, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:45:02,110-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1655bb90] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4cf6b3a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:45:02,112-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1655bb90] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4cf6b3a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:45:02,123-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [3be14d22] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:45:02,124-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [3be14d22] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:45:03,008-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ca58e03] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:45:03,008-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [ca58e03] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5507934a
>2018-05-29 18:45:03,012-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ca58e03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:45:03,013-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [ca58e03] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:45:03,016-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ca58e03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:45:03,017-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [ca58e03] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:45:03,021-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ca58e03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:45:03,022-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [ca58e03] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:45:03,024-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [ca58e03] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:45:03,027-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ca58e03] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: ca58e03, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:45:03,030-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3748253f
>2018-05-29 18:45:03,030-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:45:03,127-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:45:03,127-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:45:03,127-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:45:03,127-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:45:03,127-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3748253f
>2018-05-29 18:45:03,127-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:45:03,140-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [4c99d88d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:45:03,141-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4c99d88d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 462565f0
>2018-05-29 18:45:03,144-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4c99d88d] FINISH, SetVdsStatusVDSCommand, log id: 462565f0
>2018-05-29 18:45:03,149-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c99d88d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4c99d88d, Job ID: 4618a1e6-289c-45c4-af61-a674a0c58082, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:45:03,154-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c99d88d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3925c7f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:45:03,156-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c99d88d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3925c7f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:45:03,166-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [448ae2cb] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:45:03,167-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [448ae2cb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4a3e0424] Autorecovering 2 hosts
>2018-05-29 18:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4a3e0424] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:50:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4b9fba0f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:50:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4b9fba0f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:50:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4b9fba0f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:50:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4b9fba0f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:50:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4b9fba0f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12652ff5
>2018-05-29 18:50:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4b9fba0f] FINISH, SetVdsStatusVDSCommand, log id: 12652ff5
>2018-05-29 18:50:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4b9fba0f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:50:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4b9fba0f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:50:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4b9fba0f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:50:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [24223fbe] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [24223fbe] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [24223fbe] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [24223fbe] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:50:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [24223fbe] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d516907
>2018-05-29 18:50:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [24223fbe] FINISH, SetVdsStatusVDSCommand, log id: 2d516907
>2018-05-29 18:50:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [24223fbe] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 623001c8
>2018-05-29 18:50:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [24223fbe] FINISH, SetHaMaintenanceModeVDSCommand, log id: 623001c8
>2018-05-29 18:50:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [24223fbe] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:50:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [24223fbe] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:50:00,858-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4dab02fb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: de879be
>2018-05-29 18:50:00,859-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4dab02fb] FINISH, GetHardwareInfoVDSCommand, log id: de879be
>2018-05-29 18:50:00,872-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [69982c25] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:50:00,880-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [5885c1fb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:50:00,886-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5759b958] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:50:00,897-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5759b958] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 73ecb40d
>2018-05-29 18:50:01,271-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5759b958] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:50:01,271-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5759b958] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 73ecb40d
>2018-05-29 18:50:01,275-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5759b958] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:50:01,276-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5759b958] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:50:01,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5759b958] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:50:01,281-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5759b958] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:50:01,284-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5759b958] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:50:01,285-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5759b958] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:50:01,288-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5759b958] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:50:01,290-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5759b958] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5759b958, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:50:01,296-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1489dcca
>2018-05-29 18:50:01,296-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:50:01,392-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:50:01,392-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:50:01,392-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:50:01,392-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:50:01,392-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1489dcca
>2018-05-29 18:50:01,392-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:50:01,405-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [4c625bac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:50:01,406-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4c625bac] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 225bb82f
>2018-05-29 18:50:01,408-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4c625bac] FINISH, SetVdsStatusVDSCommand, log id: 225bb82f
>2018-05-29 18:50:01,439-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4c625bac] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4c625bac, Job ID: a123d66c-e130-475c-9b52-8c21aa3094f8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:50:01,444-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4c625bac] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5885c1fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:50:01,446-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4c625bac] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5885c1fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:50:01,457-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [df22292] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:50:01,458-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [df22292] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:50:02,754-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1b5bb008] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 8c1512e
>2018-05-29 18:50:02,756-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1b5bb008] FINISH, GetHardwareInfoVDSCommand, log id: 8c1512e
>2018-05-29 18:50:02,769-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [74a7fef1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:50:02,774-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [6d405a42] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:50:02,779-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2ac94ec2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:50:02,786-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2ac94ec2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 720adcaf
>2018-05-29 18:50:03,159-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ac94ec2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:50:03,159-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2ac94ec2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 720adcaf
>2018-05-29 18:50:03,163-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ac94ec2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:50:03,164-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2ac94ec2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:50:03,168-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ac94ec2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:50:03,168-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2ac94ec2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:50:03,172-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ac94ec2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:50:03,173-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2ac94ec2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:50:03,176-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2ac94ec2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:50:03,178-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ac94ec2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2ac94ec2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:50:03,181-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 193c0618
>2018-05-29 18:50:03,181-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:50:03,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:50:03,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:50:03,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:50:03,280-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:50:03,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 193c0618
>2018-05-29 18:50:03,280-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:50:03,293-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [2e49e900] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:50:03,295-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2e49e900] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39529b90
>2018-05-29 18:50:03,297-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2e49e900] FINISH, SetVdsStatusVDSCommand, log id: 39529b90
>2018-05-29 18:50:03,302-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e49e900] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2e49e900, Job ID: 6bf5e8c9-b488-40c9-8a1a-aa4ef6f91164, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:50:03,307-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e49e900] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6d405a42, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:50:03,309-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e49e900] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6d405a42, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:50:03,319-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [38b804ee] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:50:03,321-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [38b804ee] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:54:43,200-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler2) [731428f7] Attempting to update VMs/Templates Ovf.
>2018-05-29 18:54:43,202-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 18:54:43,202-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 18:54:43,203-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:54:43,207-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 18:54:43,210-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 18:54:43,210-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 18:54:43,210-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 18:54:43,210-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 18:54:43,211-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 18:54:43,211-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 18:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [56a8143] Autorecovering 2 hosts
>2018-05-29 18:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [56a8143] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:55:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6d1c2cce] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:55:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6d1c2cce] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:55:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6d1c2cce] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:55:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6d1c2cce] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:55:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6d1c2cce] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 477fce0f
>2018-05-29 18:55:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6d1c2cce] FINISH, SetVdsStatusVDSCommand, log id: 477fce0f
>2018-05-29 18:55:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6d1c2cce] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:55:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6d1c2cce] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 18:55:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6d1c2cce] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:55:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5263d57b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:55:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5263d57b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 18:55:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5263d57b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:55:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5263d57b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:55:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5263d57b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 478d31de
>2018-05-29 18:55:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5263d57b] FINISH, SetVdsStatusVDSCommand, log id: 478d31de
>2018-05-29 18:55:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5263d57b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 13107763
>2018-05-29 18:55:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5263d57b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 13107763
>2018-05-29 18:55:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5263d57b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 18:55:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5263d57b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 18:55:00,971-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [1009449a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 171c3eff
>2018-05-29 18:55:00,973-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [1009449a] FINISH, GetHardwareInfoVDSCommand, log id: 171c3eff
>2018-05-29 18:55:00,986-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7a7aab59] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:55:00,995-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [ce8300f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:55:01,000-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4c41d4dc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:55:01,012-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4c41d4dc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6a9c7cb7
>2018-05-29 18:55:01,391-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c41d4dc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:55:01,391-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4c41d4dc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6a9c7cb7
>2018-05-29 18:55:01,395-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c41d4dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:55:01,396-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4c41d4dc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:55:01,400-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c41d4dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:55:01,401-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4c41d4dc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:55:01,405-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c41d4dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:55:01,405-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4c41d4dc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:55:01,408-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4c41d4dc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:55:01,411-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c41d4dc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4c41d4dc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:55:01,418-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a2d7634
>2018-05-29 18:55:01,418-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:55:01,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:55:01,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:55:01,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:55:01,513-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:55:01,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a2d7634
>2018-05-29 18:55:01,513-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:55:01,527-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [43f4e3d4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:55:01,528-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [43f4e3d4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 57da54f1
>2018-05-29 18:55:01,531-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [43f4e3d4] FINISH, SetVdsStatusVDSCommand, log id: 57da54f1
>2018-05-29 18:55:01,562-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [43f4e3d4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 43f4e3d4, Job ID: 8eb7aa9d-256d-463f-96f5-8768ba183075, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:55:01,567-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [43f4e3d4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: ce8300f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:55:01,569-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [43f4e3d4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: ce8300f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:55:01,580-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [30ef83fb] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 18:55:01,581-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [30ef83fb] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 18:55:02,165-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [30d1b098] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5e41e3f0
>2018-05-29 18:55:02,167-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [30d1b098] FINISH, GetHardwareInfoVDSCommand, log id: 5e41e3f0
>2018-05-29 18:55:02,180-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [7573b7c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:55:02,184-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [35424b07] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:55:02,189-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [37ed4f66] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 18:55:02,196-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [37ed4f66] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 265a6954
>2018-05-29 18:55:03,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [37ed4f66] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 18:55:03,567-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [37ed4f66] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 265a6954
>2018-05-29 18:55:03,571-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [37ed4f66] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:55:03,572-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [37ed4f66] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:55:03,576-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [37ed4f66] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:55:03,577-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [37ed4f66] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:55:03,581-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [37ed4f66] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 18:55:03,581-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [37ed4f66] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 18:55:03,584-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [37ed4f66] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 18:55:03,587-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [37ed4f66] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 37ed4f66, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 18:55:03,590-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 40ffdcb4
>2018-05-29 18:55:03,590-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 18:55:03,686-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:55:03,686-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 18:55:03,686-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 18:55:03,686-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 18:55:03,686-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 40ffdcb4
>2018-05-29 18:55:03,686-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 18:55:03,700-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [400ac80e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:55:03,701-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [400ac80e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 27535a94
>2018-05-29 18:55:03,704-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [400ac80e] FINISH, SetVdsStatusVDSCommand, log id: 27535a94
>2018-05-29 18:55:03,709-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [400ac80e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 400ac80e, Job ID: 1b99ae7b-a248-4241-b0e0-75756d10ac78, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 18:55:03,714-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [400ac80e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 35424b07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 18:55:03,716-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [400ac80e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 35424b07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 18:55:03,727-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [5633c40c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 18:55:03,728-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [5633c40c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [606f6b78] Autorecovering 2 hosts
>2018-05-29 19:00:00,011-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [606f6b78] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:00:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c5b1cde] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c5b1cde] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c5b1cde] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c5b1cde] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:00:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6c5b1cde] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 63d8a64a
>2018-05-29 19:00:00,080-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6c5b1cde] FINISH, SetVdsStatusVDSCommand, log id: 63d8a64a
>2018-05-29 19:00:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c5b1cde] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:00:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c5b1cde] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:00:00,083-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [6c5b1cde] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:00:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [532b002d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [532b002d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [532b002d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [532b002d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:00:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [532b002d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7649cc20
>2018-05-29 19:00:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [532b002d] FINISH, SetVdsStatusVDSCommand, log id: 7649cc20
>2018-05-29 19:00:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [532b002d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 51a60e16
>2018-05-29 19:00:00,103-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [532b002d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 51a60e16
>2018-05-29 19:00:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [532b002d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:00:00,105-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [532b002d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:00:01,401-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [15bc5fdc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1a67283d
>2018-05-29 19:00:01,403-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [15bc5fdc] FINISH, GetHardwareInfoVDSCommand, log id: 1a67283d
>2018-05-29 19:00:01,417-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [3fbc6f94] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:00:01,427-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [4b92ceec] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:00:01,432-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [9ed7ce8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:00:01,503-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [9ed7ce8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4f2a5ec0
>2018-05-29 19:00:01,873-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [9ed7ce8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:00:01,873-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [9ed7ce8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4f2a5ec0
>2018-05-29 19:00:01,877-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [9ed7ce8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:00:01,878-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [9ed7ce8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:00:01,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [9ed7ce8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:00:01,882-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [9ed7ce8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:00:01,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [9ed7ce8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:00:01,887-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [9ed7ce8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:00:01,890-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [9ed7ce8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:00:01,893-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [9ed7ce8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 9ed7ce8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:00:01,899-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 196e078
>2018-05-29 19:00:01,899-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:00:01,997-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:00:01,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:00:01,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:00:01,997-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:00:01,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 196e078
>2018-05-29 19:00:01,997-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:00:02,011-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [763fed75] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:00:02,012-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [763fed75] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 10f20ec9
>2018-05-29 19:00:02,015-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [763fed75] FINISH, SetVdsStatusVDSCommand, log id: 10f20ec9
>2018-05-29 19:00:02,048-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [763fed75] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 763fed75, Job ID: e4201cea-d9c0-4992-a082-7ce70732b3f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:00:02,052-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [763fed75] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4b92ceec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:00:02,055-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [763fed75] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4b92ceec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:00:02,066-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [24fb2211] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:00:02,067-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [24fb2211] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:00:02,202-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [29bd49c0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 718c6006
>2018-05-29 19:00:02,203-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [29bd49c0] FINISH, GetHardwareInfoVDSCommand, log id: 718c6006
>2018-05-29 19:00:02,217-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [5caa9684] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:00:02,221-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [625c6dd6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:00:02,226-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [2e313e78] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:00:02,233-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [2e313e78] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 94f3c72
>2018-05-29 19:00:02,610-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2e313e78] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:00:02,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [2e313e78] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 94f3c72
>2018-05-29 19:00:02,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2e313e78] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:00:02,615-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [2e313e78] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:00:02,619-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2e313e78] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:00:02,619-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [2e313e78] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:00:02,623-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2e313e78] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:00:02,624-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [2e313e78] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:00:02,626-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [2e313e78] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:00:02,629-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2e313e78] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2e313e78, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:00:02,633-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1d036d92
>2018-05-29 19:00:02,633-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:00:02,729-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-11) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:00:02,729-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:00:02,729-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:00:02,729-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:00:02,729-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1d036d92
>2018-05-29 19:00:02,729-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-11) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:00:02,743-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [39fa1264] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:00:02,744-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [39fa1264] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 366921fa
>2018-05-29 19:00:02,746-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [39fa1264] FINISH, SetVdsStatusVDSCommand, log id: 366921fa
>2018-05-29 19:00:02,752-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [39fa1264] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 39fa1264, Job ID: 09d4cf5b-45c8-4af2-a6e5-8952e8e03c25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:00:02,757-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [39fa1264] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 625c6dd6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:00:02,760-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [39fa1264] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 625c6dd6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:00:02,771-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7460f2aa] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:00:02,772-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7460f2aa] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [38fc79f1] Autorecovering 2 hosts
>2018-05-29 19:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [38fc79f1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:05:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2859faf0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:05:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2859faf0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:05:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2859faf0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:05:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2859faf0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:05:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2859faf0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18613f68
>2018-05-29 19:05:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2859faf0] FINISH, SetVdsStatusVDSCommand, log id: 18613f68
>2018-05-29 19:05:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2859faf0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:05:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2859faf0] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:05:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2859faf0] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:05:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [647b9a8d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:05:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [647b9a8d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:05:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [647b9a8d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:05:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [647b9a8d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:05:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [647b9a8d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49c839fe
>2018-05-29 19:05:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [647b9a8d] FINISH, SetVdsStatusVDSCommand, log id: 49c839fe
>2018-05-29 19:05:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [647b9a8d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 35f569e3
>2018-05-29 19:05:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [647b9a8d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 35f569e3
>2018-05-29 19:05:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [647b9a8d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:05:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [647b9a8d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:05:00,388-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [53f816f9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7e20aec5
>2018-05-29 19:05:00,390-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [53f816f9] FINISH, GetHardwareInfoVDSCommand, log id: 7e20aec5
>2018-05-29 19:05:00,402-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [499ed0e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:05:00,411-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [21ed543a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:05:00,416-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [2076e1cb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:05:00,427-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [2076e1cb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 456565be
>2018-05-29 19:05:00,814-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2076e1cb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:05:00,814-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [2076e1cb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 456565be
>2018-05-29 19:05:00,818-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2076e1cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:05:00,819-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [2076e1cb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:05:00,823-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2076e1cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:05:00,824-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [2076e1cb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:05:00,827-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2076e1cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:05:00,828-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [2076e1cb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:05:00,831-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [2076e1cb] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:05:00,834-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2076e1cb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2076e1cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:05:00,840-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 23c734e9
>2018-05-29 19:05:00,840-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:05:00,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:05:00,940-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:05:00,940-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:05:00,940-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:05:00,940-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 23c734e9
>2018-05-29 19:05:00,940-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:05:00,954-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [4c65de4c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:05:00,955-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4c65de4c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 306642d1
>2018-05-29 19:05:00,957-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4c65de4c] FINISH, SetVdsStatusVDSCommand, log id: 306642d1
>2018-05-29 19:05:00,990-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4c65de4c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4c65de4c, Job ID: 32e304f1-d438-4b2c-894c-9f561cc499de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:05:00,995-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4c65de4c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 21ed543a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:05:00,997-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4c65de4c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 21ed543a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:05:01,008-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [40a5dfd1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:05:01,010-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [40a5dfd1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:05:02,761-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [685f3b40] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3b17f16
>2018-05-29 19:05:02,762-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [685f3b40] FINISH, GetHardwareInfoVDSCommand, log id: 3b17f16
>2018-05-29 19:05:02,776-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [3bef2dce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:05:02,780-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2389d97c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:05:02,786-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [3d3136d2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:05:02,792-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [3d3136d2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 37d68d82
>2018-05-29 19:05:03,157-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d3136d2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:05:03,157-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [3d3136d2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 37d68d82
>2018-05-29 19:05:03,161-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d3136d2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:05:03,162-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3d3136d2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:05:03,166-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d3136d2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:05:03,167-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3d3136d2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:05:03,170-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d3136d2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:05:03,171-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3d3136d2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:05:03,173-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [3d3136d2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:05:03,176-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d3136d2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3d3136d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:05:03,180-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: fc9d52
>2018-05-29 19:05:03,180-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:05:03,276-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:05:03,276-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:05:03,276-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:05:03,276-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:05:03,276-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: fc9d52
>2018-05-29 19:05:03,276-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:05:03,290-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [7c2a7214] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:05:03,291-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7c2a7214] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 77ec50df
>2018-05-29 19:05:03,293-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7c2a7214] FINISH, SetVdsStatusVDSCommand, log id: 77ec50df
>2018-05-29 19:05:03,299-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7c2a7214] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7c2a7214, Job ID: d695143f-c255-414b-9573-0fa8ce213117, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:05:03,304-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7c2a7214] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2389d97c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:05:03,306-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7c2a7214] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2389d97c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:05:03,317-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [209bc4e3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:05:03,318-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [209bc4e3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [50b91b90] Autorecovering 2 hosts
>2018-05-29 19:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [50b91b90] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:10:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29a52bda] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:10:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29a52bda] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:10:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29a52bda] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:10:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29a52bda] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:10:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [29a52bda] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 468cd578
>2018-05-29 19:10:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [29a52bda] FINISH, SetVdsStatusVDSCommand, log id: 468cd578
>2018-05-29 19:10:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29a52bda] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:10:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29a52bda] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:10:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [29a52bda] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:10:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6d623c1a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:10:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6d623c1a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:10:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6d623c1a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:10:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6d623c1a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:10:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6d623c1a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f0550cb
>2018-05-29 19:10:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6d623c1a] FINISH, SetVdsStatusVDSCommand, log id: 1f0550cb
>2018-05-29 19:10:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6d623c1a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 25231cff
>2018-05-29 19:10:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6d623c1a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 25231cff
>2018-05-29 19:10:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6d623c1a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:10:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6d623c1a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:10:01,030-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [582eee8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: b6b6c31
>2018-05-29 19:10:01,032-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [582eee8] FINISH, GetHardwareInfoVDSCommand, log id: b6b6c31
>2018-05-29 19:10:01,045-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [39787bf6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:10:01,053-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [3fa8df25] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:10:01,059-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7d86eea1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:10:01,065-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7d86eea1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2fbfca6f
>2018-05-29 19:10:01,439-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d86eea1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:10:01,439-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7d86eea1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2fbfca6f
>2018-05-29 19:10:01,443-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d86eea1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:10:01,444-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7d86eea1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:10:01,448-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d86eea1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:10:01,448-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7d86eea1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:10:01,452-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d86eea1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:10:01,453-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7d86eea1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:10:01,457-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7d86eea1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:10:01,459-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d86eea1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7d86eea1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:10:01,462-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1b1f42d1
>2018-05-29 19:10:01,462-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:10:01,562-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:10:01,562-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:10:01,562-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:10:01,562-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:10:01,562-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1b1f42d1
>2018-05-29 19:10:01,562-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:10:01,574-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [32bb3c3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:10:01,576-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [32bb3c3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f713635
>2018-05-29 19:10:01,578-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [32bb3c3] FINISH, SetVdsStatusVDSCommand, log id: 7f713635
>2018-05-29 19:10:01,583-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6d8e4cf8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 23ada0ac
>2018-05-29 19:10:01,585-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6d8e4cf8] FINISH, GetHardwareInfoVDSCommand, log id: 23ada0ac
>2018-05-29 19:10:01,598-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [17931338] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:10:01,607-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [17771cf9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:10:01,610-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [32bb3c3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 32bb3c3, Job ID: b556be66-5056-4e56-9411-1f2167735366, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:10:01,612-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7afbb651] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:10:01,615-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [32bb3c3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3fa8df25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:10:01,617-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [32bb3c3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3fa8df25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:10:01,624-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7afbb651] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 463d1b37
>2018-05-29 19:10:01,628-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [71db734c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:10:01,629-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [71db734c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:10:02,002-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7afbb651] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:10:02,002-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7afbb651] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 463d1b37
>2018-05-29 19:10:02,006-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7afbb651] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:10:02,007-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7afbb651] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:10:02,011-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7afbb651] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:10:02,011-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7afbb651] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:10:02,015-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7afbb651] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:10:02,016-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7afbb651] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:10:02,019-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7afbb651] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:10:02,022-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7afbb651] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7afbb651, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:10:02,025-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 68810767
>2018-05-29 19:10:02,025-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:10:02,119-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:10:02,119-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:10:02,119-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:10:02,119-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:10:02,119-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 68810767
>2018-05-29 19:10:02,119-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:10:02,131-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [1e90616a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:10:02,133-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1e90616a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54f1c9bf
>2018-05-29 19:10:02,135-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1e90616a] FINISH, SetVdsStatusVDSCommand, log id: 54f1c9bf
>2018-05-29 19:10:02,140-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1e90616a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1e90616a, Job ID: 0e9bd74e-ea1c-433e-acd7-7682b0602b6b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:10:02,145-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1e90616a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17771cf9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:10:02,147-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1e90616a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17771cf9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:10:02,158-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [78ead16e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:10:02,159-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [78ead16e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:15:00,006-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5a8f0ef5] Autorecovering 2 hosts
>2018-05-29 19:15:00,006-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5a8f0ef5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:15:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3698609c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:15:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3698609c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:15:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3698609c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:15:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3698609c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:15:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3698609c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 496dbbe6
>2018-05-29 19:15:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3698609c] FINISH, SetVdsStatusVDSCommand, log id: 496dbbe6
>2018-05-29 19:15:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3698609c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:15:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3698609c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:15:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3698609c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:15:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7af6eff5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:15:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7af6eff5] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:15:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7af6eff5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:15:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7af6eff5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:15:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7af6eff5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61546a1e
>2018-05-29 19:15:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7af6eff5] FINISH, SetVdsStatusVDSCommand, log id: 61546a1e
>2018-05-29 19:15:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7af6eff5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3e33a128
>2018-05-29 19:15:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7af6eff5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3e33a128
>2018-05-29 19:15:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7af6eff5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:15:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7af6eff5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:15:02,373-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [58128c5b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2dbb0334
>2018-05-29 19:15:02,375-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [58128c5b] FINISH, GetHardwareInfoVDSCommand, log id: 2dbb0334
>2018-05-29 19:15:02,388-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [5aead7a3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:15:02,397-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [4fbc6be9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:15:02,402-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [21518a59] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:15:02,413-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [21518a59] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 72cd1d51
>2018-05-29 19:15:02,758-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1c87dfd3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5b959942
>2018-05-29 19:15:02,760-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1c87dfd3] FINISH, GetHardwareInfoVDSCommand, log id: 5b959942
>2018-05-29 19:15:02,773-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [6ad8a93b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:15:02,777-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [41cd32b6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:15:02,782-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7117d6c4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:15:02,788-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7117d6c4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4a760fcc
>2018-05-29 19:15:03,169-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7117d6c4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:15:03,169-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7117d6c4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4a760fcc
>2018-05-29 19:15:03,173-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7117d6c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:15:03,174-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7117d6c4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:15:03,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7117d6c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:15:03,178-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7117d6c4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:15:03,182-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7117d6c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:15:03,183-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7117d6c4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:15:03,185-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7117d6c4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:15:03,188-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7117d6c4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7117d6c4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:15:03,194-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 781a4a78
>2018-05-29 19:15:03,194-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:15:03,291-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:15:03,291-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:15:03,291-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:15:03,291-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:15:03,291-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 781a4a78
>2018-05-29 19:15:03,291-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:15:03,305-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [65451069] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:15:03,306-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [65451069] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 518b628b
>2018-05-29 19:15:03,308-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [65451069] FINISH, SetVdsStatusVDSCommand, log id: 518b628b
>2018-05-29 19:15:03,340-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [65451069] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 65451069, Job ID: 2cce2ed7-db44-4bde-a775-dfbccf935333, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:15:03,345-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [65451069] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 41cd32b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:15:03,347-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [65451069] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 41cd32b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:15:03,358-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [48d0c8a0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:15:03,360-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [48d0c8a0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:15:03,782-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21518a59] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:15:03,782-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [21518a59] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 72cd1d51
>2018-05-29 19:15:03,785-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21518a59] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:15:03,786-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [21518a59] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:15:03,790-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21518a59] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:15:03,790-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [21518a59] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:15:03,794-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21518a59] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:15:03,795-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [21518a59] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:15:03,798-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [21518a59] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:15:03,801-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21518a59] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 21518a59, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:15:03,803-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 29b36b22
>2018-05-29 19:15:03,804-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:15:03,899-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:15:03,899-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:15:03,899-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:15:03,900-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:15:03,900-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 29b36b22
>2018-05-29 19:15:03,900-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:15:03,913-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [3c92283] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:15:03,915-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3c92283] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12951b95
>2018-05-29 19:15:03,917-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3c92283] FINISH, SetVdsStatusVDSCommand, log id: 12951b95
>2018-05-29 19:15:03,922-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c92283] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c92283, Job ID: a465a10e-692e-48d6-b91f-f141d06fb899, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:15:03,927-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c92283] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4fbc6be9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:15:03,929-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c92283] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4fbc6be9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:15:03,939-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [4779a7ed] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:15:03,940-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4779a7ed] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:20:00,011-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [ea53f73] Autorecovering 2 hosts
>2018-05-29 19:20:00,011-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [ea53f73] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:20:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [826cee1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:20:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [826cee1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:20:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [826cee1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:20:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [826cee1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:20:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [826cee1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b3d4c03
>2018-05-29 19:20:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [826cee1] FINISH, SetVdsStatusVDSCommand, log id: 1b3d4c03
>2018-05-29 19:20:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [826cee1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:20:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [826cee1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:20:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [826cee1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:20:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2d26fdd6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:20:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2d26fdd6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:20:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2d26fdd6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:20:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2d26fdd6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:20:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2d26fdd6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b1d3adf
>2018-05-29 19:20:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2d26fdd6] FINISH, SetVdsStatusVDSCommand, log id: 1b1d3adf
>2018-05-29 19:20:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [2d26fdd6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 34e7f64d
>2018-05-29 19:20:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [2d26fdd6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 34e7f64d
>2018-05-29 19:20:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2d26fdd6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:20:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2d26fdd6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:20:01,056-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3cb95ba6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3ce7a1c
>2018-05-29 19:20:01,058-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3cb95ba6] FINISH, GetHardwareInfoVDSCommand, log id: 3ce7a1c
>2018-05-29 19:20:01,071-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [b5f2a2e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:20:01,080-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [24c3705b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:20:01,086-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [70ba9239] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:20:01,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [70ba9239] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 30b03270
>2018-05-29 19:20:01,474-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [70ba9239] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:20:01,474-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [70ba9239] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 30b03270
>2018-05-29 19:20:01,478-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [70ba9239] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:20:01,479-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [70ba9239] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:20:01,483-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [70ba9239] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:20:01,484-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [70ba9239] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:20:01,488-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [70ba9239] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:20:01,489-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [70ba9239] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:20:01,493-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [70ba9239] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:20:01,495-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [70ba9239] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 70ba9239, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:20:01,502-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4080a147
>2018-05-29 19:20:01,502-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:20:01,563-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [318d0c04] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2daff476
>2018-05-29 19:20:01,565-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [318d0c04] FINISH, GetHardwareInfoVDSCommand, log id: 2daff476
>2018-05-29 19:20:01,577-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [51a5d531] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:20:01,582-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [693071c0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:20:01,587-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [10a50851] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:20:01,593-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [10a50851] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 216c0481
>2018-05-29 19:20:01,601-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:20:01,601-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:20:01,601-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:20:01,601-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:20:01,601-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4080a147
>2018-05-29 19:20:01,601-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:20:01,614-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [18cf21dd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:20:01,615-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [18cf21dd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f29f40f
>2018-05-29 19:20:01,617-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [18cf21dd] FINISH, SetVdsStatusVDSCommand, log id: 5f29f40f
>2018-05-29 19:20:01,648-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [18cf21dd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 18cf21dd, Job ID: 490af0d1-71ee-47a0-9fac-28591756f23d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:20:01,653-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [18cf21dd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 24c3705b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:20:01,655-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [18cf21dd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 24c3705b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:20:01,666-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [e7922d2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:20:01,667-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [e7922d2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:20:01,963-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [10a50851] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:20:01,963-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [10a50851] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 216c0481
>2018-05-29 19:20:01,968-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [10a50851] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:20:01,968-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [10a50851] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:20:01,972-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [10a50851] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:20:01,973-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [10a50851] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:20:01,977-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [10a50851] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:20:01,977-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [10a50851] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:20:01,980-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [10a50851] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:20:01,983-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [10a50851] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 10a50851, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:20:01,986-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2f9564e9
>2018-05-29 19:20:01,986-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:20:02,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:20:02,082-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:20:02,082-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:20:02,082-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:20:02,082-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2f9564e9
>2018-05-29 19:20:02,082-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:20:02,094-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [22d2a8a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:20:02,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [22d2a8a6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1aab1cdc
>2018-05-29 19:20:02,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [22d2a8a6] FINISH, SetVdsStatusVDSCommand, log id: 1aab1cdc
>2018-05-29 19:20:02,103-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [22d2a8a6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22d2a8a6, Job ID: 3605d5f3-8666-4975-94c1-e87edc49a94b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:20:02,107-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [22d2a8a6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 693071c0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:20:02,109-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [22d2a8a6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 693071c0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:20:02,119-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [7b964f5f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:20:02,120-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [7b964f5f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:25:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [73c0daaf] Autorecovering 2 hosts
>2018-05-29 19:25:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [73c0daaf] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:25:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29695546] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:25:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29695546] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:25:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29695546] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:25:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29695546] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:25:00,045-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [29695546] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5888efdf
>2018-05-29 19:25:00,048-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [29695546] FINISH, SetVdsStatusVDSCommand, log id: 5888efdf
>2018-05-29 19:25:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29695546] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:25:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29695546] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:25:00,050-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [29695546] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:25:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e0feddc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:25:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e0feddc] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:25:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e0feddc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:25:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e0feddc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:25:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6e0feddc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5932db8c
>2018-05-29 19:25:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6e0feddc] FINISH, SetVdsStatusVDSCommand, log id: 5932db8c
>2018-05-29 19:25:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6e0feddc] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 30baa6c0
>2018-05-29 19:25:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6e0feddc] FINISH, SetHaMaintenanceModeVDSCommand, log id: 30baa6c0
>2018-05-29 19:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e0feddc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:25:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e0feddc] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:25:02,241-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5fb17a3f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5fed4270
>2018-05-29 19:25:02,243-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5fb17a3f] FINISH, GetHardwareInfoVDSCommand, log id: 5fed4270
>2018-05-29 19:25:02,255-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [5163404f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:25:02,260-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [537d3261] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:25:02,265-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4f0e63b4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:25:02,277-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4f0e63b4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 18cb99fb
>2018-05-29 19:25:02,656-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f0e63b4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:25:02,656-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4f0e63b4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 18cb99fb
>2018-05-29 19:25:02,660-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f0e63b4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:25:02,661-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4f0e63b4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:25:02,665-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f0e63b4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:25:02,666-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4f0e63b4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:25:02,670-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f0e63b4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:25:02,671-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4f0e63b4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:25:02,673-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4f0e63b4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:25:02,676-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f0e63b4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4f0e63b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:25:02,682-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7a0fe7e4
>2018-05-29 19:25:02,682-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:25:02,780-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:25:02,780-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:25:02,780-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:25:02,780-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:25:02,780-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7a0fe7e4
>2018-05-29 19:25:02,780-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:25:02,793-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [5a61b335] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:25:02,794-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5a61b335] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7ff116c8
>2018-05-29 19:25:02,796-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5a61b335] FINISH, SetVdsStatusVDSCommand, log id: 7ff116c8
>2018-05-29 19:25:02,805-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5a61b335] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5a61b335, Job ID: 7870d9da-df49-41d0-ac73-b82be5c8d638, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:25:02,809-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5a61b335] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 537d3261, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:25:02,813-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2fde64d3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5f7370c2
>2018-05-29 19:25:02,813-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5a61b335] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 537d3261, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:25:02,814-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2fde64d3] FINISH, GetHardwareInfoVDSCommand, log id: 5f7370c2
>2018-05-29 19:25:02,826-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [3ef076ed] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:25:02,827-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [3ef076ed] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:25:02,828-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [27cb2639] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:25:02,832-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [257cbb4e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:25:02,837-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [109a9f5c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:25:02,844-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [109a9f5c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 626c78ce
>2018-05-29 19:25:04,211-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [109a9f5c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:25:04,211-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [109a9f5c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 626c78ce
>2018-05-29 19:25:04,215-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [109a9f5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:25:04,215-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [109a9f5c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:25:04,219-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [109a9f5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:25:04,220-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [109a9f5c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:25:04,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [109a9f5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:25:04,225-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [109a9f5c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:25:04,229-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [109a9f5c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:25:04,231-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [109a9f5c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 109a9f5c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:25:04,234-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 491862b7
>2018-05-29 19:25:04,234-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:25:04,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:25:04,332-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:25:04,332-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:25:04,332-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:25:04,332-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 491862b7
>2018-05-29 19:25:04,332-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:25:04,345-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [4b32fe28] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:25:04,346-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4b32fe28] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2edff71a
>2018-05-29 19:25:04,349-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4b32fe28] FINISH, SetVdsStatusVDSCommand, log id: 2edff71a
>2018-05-29 19:25:04,354-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4b32fe28] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4b32fe28, Job ID: e9f62954-21c1-4ee8-9cef-29a5844e68c6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:25:04,359-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4b32fe28] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 257cbb4e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:25:04,361-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4b32fe28] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 257cbb4e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:25:04,371-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [7e1674ff] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:25:04,372-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [7e1674ff] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [368672dd] Autorecovering 2 hosts
>2018-05-29 19:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [368672dd] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:30:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2405589f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:30:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2405589f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:30:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2405589f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:30:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2405589f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:30:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2405589f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 535f7d40
>2018-05-29 19:30:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2405589f] FINISH, SetVdsStatusVDSCommand, log id: 535f7d40
>2018-05-29 19:30:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2405589f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:30:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2405589f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:30:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2405589f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:30:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [20e22f6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:30:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [20e22f6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:30:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [20e22f6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:30:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [20e22f6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:30:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [20e22f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7abc4c18
>2018-05-29 19:30:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [20e22f6] FINISH, SetVdsStatusVDSCommand, log id: 7abc4c18
>2018-05-29 19:30:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [20e22f6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: c1b57b4
>2018-05-29 19:30:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [20e22f6] FINISH, SetHaMaintenanceModeVDSCommand, log id: c1b57b4
>2018-05-29 19:30:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [20e22f6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:30:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [20e22f6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:30:00,415-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5040bb37] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1f1b6ec8
>2018-05-29 19:30:00,417-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5040bb37] FINISH, GetHardwareInfoVDSCommand, log id: 1f1b6ec8
>2018-05-29 19:30:00,429-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [2f6672d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:30:00,437-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [6b1fc73c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:30:00,442-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [18ca1a8f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:30:00,454-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [18ca1a8f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5f686f04
>2018-05-29 19:30:00,833-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ca1a8f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:30:00,833-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [18ca1a8f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5f686f04
>2018-05-29 19:30:00,837-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ca1a8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:30:00,837-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [18ca1a8f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:30:00,841-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ca1a8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:30:00,841-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [18ca1a8f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:30:00,845-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ca1a8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:30:00,846-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [18ca1a8f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:30:00,848-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [18ca1a8f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:30:00,851-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ca1a8f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 18ca1a8f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:30:00,858-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 67d6c82c
>2018-05-29 19:30:00,858-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:30:00,957-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-10) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:30:00,957-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:30:00,957-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:30:00,957-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:30:00,957-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] FINISH, ConnectStoragePoolVDSCommand, log id: 67d6c82c
>2018-05-29 19:30:00,957-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-10) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:30:00,970-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [1381138c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:30:00,971-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1381138c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3917a64
>2018-05-29 19:30:00,973-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1381138c] FINISH, SetVdsStatusVDSCommand, log id: 3917a64
>2018-05-29 19:30:01,004-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1381138c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1381138c, Job ID: 4e8a1d35-8eb7-459f-8d1e-faca59b1417f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:30:01,009-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1381138c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6b1fc73c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:30:01,011-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1381138c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6b1fc73c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:30:01,022-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [580e738e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:30:01,023-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [580e738e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:30:02,050-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7b8648c3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 38b40b34
>2018-05-29 19:30:02,052-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7b8648c3] FINISH, GetHardwareInfoVDSCommand, log id: 38b40b34
>2018-05-29 19:30:02,065-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [6b88bea6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:30:02,069-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [621c81ea] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:30:02,074-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [61b87e8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:30:02,081-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [61b87e8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 33c910e7
>2018-05-29 19:30:02,449-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [61b87e8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:30:02,449-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [61b87e8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 33c910e7
>2018-05-29 19:30:02,452-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [61b87e8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:30:02,453-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [61b87e8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:30:02,457-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [61b87e8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:30:02,458-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [61b87e8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:30:02,461-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [61b87e8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:30:02,462-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [61b87e8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:30:02,465-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [61b87e8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:30:02,467-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [61b87e8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 61b87e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:30:02,470-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 412b719d
>2018-05-29 19:30:02,470-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:30:02,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:30:02,567-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:30:02,567-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:30:02,567-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:30:02,567-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 412b719d
>2018-05-29 19:30:02,567-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:30:02,580-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [6bdf0df4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:30:02,581-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6bdf0df4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45f26b87
>2018-05-29 19:30:02,583-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6bdf0df4] FINISH, SetVdsStatusVDSCommand, log id: 45f26b87
>2018-05-29 19:30:02,589-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bdf0df4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6bdf0df4, Job ID: 92ad1f1f-e945-466d-9818-38e1dc4fb2d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:30:02,593-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bdf0df4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 621c81ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:30:02,595-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bdf0df4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 621c81ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:30:02,606-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [4f5c866] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:30:02,607-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [4f5c866] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:35:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [93e2fe4] Autorecovering 2 hosts
>2018-05-29 19:35:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [93e2fe4] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:35:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [c08c658] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:35:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [c08c658] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:35:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [c08c658] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:35:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [c08c658] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:35:00,047-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [c08c658] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 124b159c
>2018-05-29 19:35:00,050-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [c08c658] FINISH, SetVdsStatusVDSCommand, log id: 124b159c
>2018-05-29 19:35:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [c08c658] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:35:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [c08c658] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:35:00,052-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [c08c658] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:35:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22885f6f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:35:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22885f6f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:35:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22885f6f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:35:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22885f6f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:35:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [22885f6f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7da05cda
>2018-05-29 19:35:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [22885f6f] FINISH, SetVdsStatusVDSCommand, log id: 7da05cda
>2018-05-29 19:35:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [22885f6f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4561c8a4
>2018-05-29 19:35:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [22885f6f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4561c8a4
>2018-05-29 19:35:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22885f6f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:35:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22885f6f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:35:00,298-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5bd79783] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1e61a668
>2018-05-29 19:35:00,299-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5bd79783] FINISH, GetHardwareInfoVDSCommand, log id: 1e61a668
>2018-05-29 19:35:00,312-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [39ab223a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:35:00,317-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [661ef0ab] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:35:00,322-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [74c7078a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:35:00,329-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [74c7078a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 34ea7188
>2018-05-29 19:35:00,707-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74c7078a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:35:00,707-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [74c7078a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 34ea7188
>2018-05-29 19:35:00,712-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74c7078a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:35:00,713-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [74c7078a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:35:00,717-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74c7078a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:35:00,717-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [74c7078a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:35:00,722-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74c7078a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:35:00,722-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [74c7078a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:35:00,725-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [74c7078a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:35:00,728-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74c7078a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 74c7078a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:35:00,731-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7c4d67a2
>2018-05-29 19:35:00,731-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:35:00,834-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:35:00,834-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:35:00,834-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:35:00,834-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:35:00,834-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7c4d67a2
>2018-05-29 19:35:00,834-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:35:00,847-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [120e8ef6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:35:00,848-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [120e8ef6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: db6ee97
>2018-05-29 19:35:00,851-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [120e8ef6] FINISH, SetVdsStatusVDSCommand, log id: db6ee97
>2018-05-29 19:35:00,881-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [120e8ef6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 120e8ef6, Job ID: 735a91b5-f6ab-440a-a68f-caba8bc85a99, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:35:00,886-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [120e8ef6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 661ef0ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:35:00,888-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [120e8ef6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 661ef0ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:35:00,899-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [6e862d4c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:35:00,900-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [6e862d4c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:35:01,663-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7645d898] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 14399609
>2018-05-29 19:35:01,665-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7645d898] FINISH, GetHardwareInfoVDSCommand, log id: 14399609
>2018-05-29 19:35:01,677-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [f8bd024] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:35:01,681-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [109ddad2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:35:01,686-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2ecda47f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:35:01,692-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2ecda47f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e3e2157
>2018-05-29 19:35:02,073-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ecda47f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:35:02,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2ecda47f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e3e2157
>2018-05-29 19:35:02,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ecda47f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:35:02,078-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2ecda47f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:35:02,082-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ecda47f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:35:02,083-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2ecda47f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:35:02,087-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ecda47f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:35:02,087-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2ecda47f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:35:02,090-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2ecda47f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:35:02,093-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ecda47f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2ecda47f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:35:02,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4f8e9b80
>2018-05-29 19:35:02,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:35:02,192-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:35:02,192-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:35:02,192-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:35:02,192-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:35:02,192-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4f8e9b80
>2018-05-29 19:35:02,192-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:35:02,205-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [ee66b56] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:35:02,206-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [ee66b56] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76209c88
>2018-05-29 19:35:02,208-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [ee66b56] FINISH, SetVdsStatusVDSCommand, log id: 76209c88
>2018-05-29 19:35:02,214-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [ee66b56] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: ee66b56, Job ID: 4585b07d-48bd-4c1b-9159-950e9c0eb15a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:35:02,218-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [ee66b56] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 109ddad2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:35:02,221-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [ee66b56] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 109ddad2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:35:02,232-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [2f633630] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:35:02,233-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [2f633630] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:40:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [706b57d8] Autorecovering 2 hosts
>2018-05-29 19:40:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [706b57d8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:40:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [bd5d5ca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:40:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [bd5d5ca] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:40:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [bd5d5ca] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:40:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [bd5d5ca] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:40:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [bd5d5ca] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 69782d42
>2018-05-29 19:40:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [bd5d5ca] FINISH, SetVdsStatusVDSCommand, log id: 69782d42
>2018-05-29 19:40:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [bd5d5ca] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:40:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [bd5d5ca] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:40:00,080-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [bd5d5ca] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:40:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [21bbb851] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:40:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [21bbb851] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:40:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [21bbb851] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:40:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [21bbb851] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:40:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [21bbb851] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b8cfaeb
>2018-05-29 19:40:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [21bbb851] FINISH, SetVdsStatusVDSCommand, log id: 3b8cfaeb
>2018-05-29 19:40:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [21bbb851] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4f03562c
>2018-05-29 19:40:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [21bbb851] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4f03562c
>2018-05-29 19:40:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [21bbb851] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:40:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [21bbb851] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:40:01,628-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5d9970c5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 73316bb8
>2018-05-29 19:40:01,630-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5d9970c5] FINISH, GetHardwareInfoVDSCommand, log id: 73316bb8
>2018-05-29 19:40:01,643-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [7f36373] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:40:01,651-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3165446a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:40:01,656-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7a294c88] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:40:01,667-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7a294c88] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6dd32c3
>2018-05-29 19:40:02,034-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a294c88] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:40:02,034-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7a294c88] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6dd32c3
>2018-05-29 19:40:02,038-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a294c88] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:40:02,038-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7a294c88] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:40:02,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a294c88] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:40:02,043-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7a294c88] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:40:02,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a294c88] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:40:02,048-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7a294c88] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:40:02,050-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7a294c88] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:40:02,053-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a294c88] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7a294c88, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:40:02,059-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 410fd28c
>2018-05-29 19:40:02,059-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:40:02,155-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:40:02,155-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:40:02,155-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:40:02,155-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:40:02,155-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 410fd28c
>2018-05-29 19:40:02,155-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:40:02,169-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [2980f495] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:40:02,170-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2980f495] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 269df67c
>2018-05-29 19:40:02,172-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2980f495] FINISH, SetVdsStatusVDSCommand, log id: 269df67c
>2018-05-29 19:40:02,203-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2980f495] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2980f495, Job ID: 69c88811-a665-4035-9ece-d7f4d6b8af8d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:40:02,207-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2980f495] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3165446a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:40:02,209-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2980f495] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3165446a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:40:02,220-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [c9d64ad] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:40:02,221-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [c9d64ad] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:40:03,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4b06b282] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 490acd2f
>2018-05-29 19:40:03,078-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4b06b282] FINISH, GetHardwareInfoVDSCommand, log id: 490acd2f
>2018-05-29 19:40:03,090-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [5c5b9666] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:40:03,094-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [30292922] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:40:03,100-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [313075f5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:40:03,106-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [313075f5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 34cbb8b5
>2018-05-29 19:40:03,488-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [313075f5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:40:03,488-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [313075f5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 34cbb8b5
>2018-05-29 19:40:03,492-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [313075f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:40:03,493-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [313075f5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:40:03,496-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [313075f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:40:03,497-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [313075f5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:40:03,501-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [313075f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:40:03,502-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [313075f5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:40:03,504-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [313075f5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:40:03,507-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [313075f5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 313075f5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:40:03,510-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 18195307
>2018-05-29 19:40:03,510-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:40:03,605-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:40:03,605-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:40:03,605-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:40:03,605-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:40:03,605-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 18195307
>2018-05-29 19:40:03,605-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:40:03,618-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [7b47c4b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:40:03,620-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7b47c4b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4d42da48
>2018-05-29 19:40:03,622-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7b47c4b] FINISH, SetVdsStatusVDSCommand, log id: 4d42da48
>2018-05-29 19:40:03,627-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7b47c4b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7b47c4b, Job ID: 190056cb-55a8-4174-8297-5e30ebc88b12, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:40:03,632-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7b47c4b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 30292922, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:40:03,634-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7b47c4b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 30292922, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:40:03,645-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [3e042be4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:40:03,646-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [3e042be4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [70f6c26] Autorecovering 2 hosts
>2018-05-29 19:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [70f6c26] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:45:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6b5a6b4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6b5a6b4] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6b5a6b4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6b5a6b4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:45:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6b5a6b4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 663a88b9
>2018-05-29 19:45:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6b5a6b4] FINISH, SetVdsStatusVDSCommand, log id: 663a88b9
>2018-05-29 19:45:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6b5a6b4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:45:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6b5a6b4] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:45:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6b5a6b4] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:45:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1330353b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:45:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1330353b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:45:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1330353b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:45:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1330353b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:45:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1330353b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 71c589b0
>2018-05-29 19:45:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1330353b] FINISH, SetVdsStatusVDSCommand, log id: 71c589b0
>2018-05-29 19:45:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [1330353b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3e228695
>2018-05-29 19:45:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [1330353b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3e228695
>2018-05-29 19:45:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1330353b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:45:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1330353b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:45:01,283-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7c8860f7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2a5cd2e7
>2018-05-29 19:45:01,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7c8860f7] FINISH, GetHardwareInfoVDSCommand, log id: 2a5cd2e7
>2018-05-29 19:45:01,297-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [508e96f0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:45:01,305-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [47d15cc6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:45:01,310-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2ca5b686] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:45:01,322-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2ca5b686] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: ccad2ba
>2018-05-29 19:45:01,701-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2ca5b686] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:45:01,701-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2ca5b686] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: ccad2ba
>2018-05-29 19:45:01,705-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2ca5b686] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:45:01,705-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2ca5b686] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:45:01,709-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2ca5b686] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:45:01,710-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2ca5b686] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:45:01,714-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2ca5b686] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:45:01,714-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2ca5b686] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:45:01,717-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2ca5b686] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:45:01,720-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2ca5b686] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2ca5b686, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:45:01,727-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 65d2a824
>2018-05-29 19:45:01,727-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:45:01,825-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:45:01,825-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:45:01,825-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:45:01,825-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:45:01,825-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 65d2a824
>2018-05-29 19:45:01,825-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:45:01,837-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [219713ce] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:45:01,839-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [219713ce] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b4a16bd
>2018-05-29 19:45:01,841-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [219713ce] FINISH, SetVdsStatusVDSCommand, log id: 4b4a16bd
>2018-05-29 19:45:01,872-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [219713ce] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 219713ce, Job ID: 7f3fbacb-60e2-4c40-870b-a3947489ed2b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:45:01,876-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [219713ce] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 47d15cc6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:45:01,878-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [219713ce] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 47d15cc6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:45:01,889-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [765f012f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:45:01,890-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [765f012f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:45:02,917-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7011f47] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6db783a
>2018-05-29 19:45:02,919-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7011f47] FINISH, GetHardwareInfoVDSCommand, log id: 6db783a
>2018-05-29 19:45:02,931-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [184c41cf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:45:02,936-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [75b66999] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:45:02,941-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7f40d0b3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:45:02,947-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7f40d0b3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 42ddb6a5
>2018-05-29 19:45:04,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7f40d0b3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:45:04,327-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7f40d0b3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 42ddb6a5
>2018-05-29 19:45:04,331-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7f40d0b3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:45:04,332-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7f40d0b3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:45:04,336-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7f40d0b3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:45:04,337-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7f40d0b3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:45:04,341-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7f40d0b3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:45:04,341-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7f40d0b3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:45:04,344-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7f40d0b3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:45:04,347-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7f40d0b3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7f40d0b3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:45:04,349-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 34a0b062
>2018-05-29 19:45:04,350-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:45:06,396-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:45:06,396-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:45:06,396-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:45:06,396-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:45:06,396-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 34a0b062
>2018-05-29 19:45:06,396-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:45:06,409-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [24162269] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:45:06,411-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [24162269] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 533b43ae
>2018-05-29 19:45:06,413-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [24162269] FINISH, SetVdsStatusVDSCommand, log id: 533b43ae
>2018-05-29 19:45:06,445-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24162269] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 24162269, Job ID: 41c30954-2a24-4a1a-80d1-040f176c8f9a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:45:06,449-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24162269] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 75b66999, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:45:06,452-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24162269] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 75b66999, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:45:06,462-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [621a08ea] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:45:06,463-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [621a08ea] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [1b70da4a] Autorecovering 2 hosts
>2018-05-29 19:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [1b70da4a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:50:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [695b03e8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:50:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [695b03e8] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:50:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [695b03e8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:50:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [695b03e8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:50:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [695b03e8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c71af79
>2018-05-29 19:50:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [695b03e8] FINISH, SetVdsStatusVDSCommand, log id: 7c71af79
>2018-05-29 19:50:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [695b03e8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:50:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [695b03e8] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:50:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [695b03e8] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:50:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1950beae] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1950beae] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1950beae] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1950beae] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:50:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1950beae] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 15afcba1
>2018-05-29 19:50:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1950beae] FINISH, SetVdsStatusVDSCommand, log id: 15afcba1
>2018-05-29 19:50:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [1950beae] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 500ed6cb
>2018-05-29 19:50:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [1950beae] FINISH, SetHaMaintenanceModeVDSCommand, log id: 500ed6cb
>2018-05-29 19:50:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1950beae] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:50:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1950beae] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:50:01,086-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [28672917] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6725309d
>2018-05-29 19:50:01,088-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [28672917] FINISH, GetHardwareInfoVDSCommand, log id: 6725309d
>2018-05-29 19:50:01,101-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3b8f016a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:50:01,109-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1eb53a63] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:50:01,114-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7d2eb441] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:50:01,168-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7d2eb441] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 57611ca6
>2018-05-29 19:50:02,500-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6d0d4b38] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 8186499
>2018-05-29 19:50:02,502-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6d0d4b38] FINISH, GetHardwareInfoVDSCommand, log id: 8186499
>2018-05-29 19:50:02,515-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [866fbfd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:50:02,519-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [68e83ef7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:50:02,524-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [24f340ac] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:50:02,530-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [24f340ac] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 471fd7f
>2018-05-29 19:50:02,906-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24f340ac] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:50:02,906-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [24f340ac] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 471fd7f
>2018-05-29 19:50:02,911-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24f340ac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:50:02,911-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [24f340ac] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:50:02,915-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24f340ac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:50:02,915-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [24f340ac] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:50:02,919-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24f340ac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:50:02,920-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [24f340ac] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:50:02,922-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [24f340ac] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:50:02,925-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24f340ac] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 24f340ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:50:02,931-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3962c9b0
>2018-05-29 19:50:02,931-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:50:03,029-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:50:03,029-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:50:03,029-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:50:03,029-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:50:03,029-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3962c9b0
>2018-05-29 19:50:03,029-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:50:03,042-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [148242ab] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:50:03,043-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [148242ab] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b5114a5
>2018-05-29 19:50:03,045-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [148242ab] FINISH, SetVdsStatusVDSCommand, log id: 3b5114a5
>2018-05-29 19:50:03,078-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [148242ab] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 148242ab, Job ID: a8ead82a-9beb-4afb-95b4-d54cec13efdb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:50:03,082-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [148242ab] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 68e83ef7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:50:03,084-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [148242ab] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 68e83ef7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:50:03,094-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5536dc17] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:50:03,095-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5536dc17] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:50:03,477-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7d2eb441] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:50:03,477-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7d2eb441] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 57611ca6
>2018-05-29 19:50:03,481-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7d2eb441] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:50:03,481-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7d2eb441] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:50:03,485-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7d2eb441] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:50:03,486-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7d2eb441] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:50:03,489-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7d2eb441] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:50:03,490-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7d2eb441] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:50:03,493-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7d2eb441] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:50:03,495-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7d2eb441] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7d2eb441, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:50:03,498-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 35720c58
>2018-05-29 19:50:03,498-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:50:03,596-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:50:03,596-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:50:03,596-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:50:03,596-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:50:03,596-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 35720c58
>2018-05-29 19:50:03,596-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:50:03,608-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [267a6248] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:50:03,609-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [267a6248] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12454b8
>2018-05-29 19:50:03,611-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [267a6248] FINISH, SetVdsStatusVDSCommand, log id: 12454b8
>2018-05-29 19:50:03,616-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [267a6248] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 267a6248, Job ID: 4d825870-5fa7-46b4-9819-ee94ceeb389f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:50:03,620-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [267a6248] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1eb53a63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:50:03,622-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [267a6248] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1eb53a63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:50:03,632-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [2a4b250] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:50:03,633-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [2a4b250] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:54:38,360-04 INFO  [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler2) [bbcd86b] Backup check started.
>2018-05-29 19:54:38,373-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bbcd86b] EVENT_ID: ENGINE_NO_FULL_BACKUP(9,022), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: There is no full backup available, please run engine-backup to prevent data loss in case of corruption.
>2018-05-29 19:54:38,373-04 INFO  [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler2) [bbcd86b] Backup check completed.
>2018-05-29 19:54:43,211-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler7) [1956c667] Attempting to update VMs/Templates Ovf.
>2018-05-29 19:54:43,214-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 19:54:43,214-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 19:54:43,215-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:54:43,261-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 19:54:43,265-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 19:54:43,265-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 19:54:43,265-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 19:54:43,265-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 19:54:43,265-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 19:54:43,266-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 19:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1beabcec] Autorecovering 2 hosts
>2018-05-29 19:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1beabcec] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:55:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146d95b8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:55:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146d95b8] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:55:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146d95b8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:55:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146d95b8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:55:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [146d95b8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 541d522
>2018-05-29 19:55:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [146d95b8] FINISH, SetVdsStatusVDSCommand, log id: 541d522
>2018-05-29 19:55:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146d95b8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:55:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146d95b8] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 19:55:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [146d95b8] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:55:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [74f9d2d6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:55:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [74f9d2d6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 19:55:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [74f9d2d6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:55:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [74f9d2d6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:55:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [74f9d2d6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61456371
>2018-05-29 19:55:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [74f9d2d6] FINISH, SetVdsStatusVDSCommand, log id: 61456371
>2018-05-29 19:55:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [74f9d2d6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 33a2583d
>2018-05-29 19:55:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [74f9d2d6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 33a2583d
>2018-05-29 19:55:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [74f9d2d6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 19:55:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [74f9d2d6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 19:55:00,675-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [47bb41ac] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 750660f5
>2018-05-29 19:55:00,677-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [47bb41ac] FINISH, GetHardwareInfoVDSCommand, log id: 750660f5
>2018-05-29 19:55:00,690-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [209d1580] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:55:00,698-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [61a992cc] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:55:00,703-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [7f8bec9f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:55:00,715-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [7f8bec9f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2e6f2a48
>2018-05-29 19:55:01,101-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f8bec9f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:55:01,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [7f8bec9f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2e6f2a48
>2018-05-29 19:55:01,105-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f8bec9f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:55:01,105-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7f8bec9f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:55:01,109-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f8bec9f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:55:01,109-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7f8bec9f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:55:01,113-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f8bec9f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:55:01,114-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7f8bec9f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:55:01,116-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [7f8bec9f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:55:01,119-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f8bec9f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7f8bec9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:55:01,125-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7c0f7dd8
>2018-05-29 19:55:01,125-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:55:01,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:55:01,224-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:55:01,224-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:55:01,224-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:55:01,224-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7c0f7dd8
>2018-05-29 19:55:01,224-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:55:01,237-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [5c4a5a5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:55:01,238-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5c4a5a5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3cf52768
>2018-05-29 19:55:01,241-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5c4a5a5] FINISH, SetVdsStatusVDSCommand, log id: 3cf52768
>2018-05-29 19:55:01,272-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c4a5a5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5c4a5a5, Job ID: 9e343e0b-2989-4bc9-a947-2385b5c3c789, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:55:01,276-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c4a5a5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 61a992cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:55:01,278-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c4a5a5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 61a992cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:55:01,288-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1e8df2d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 45ed8dc4
>2018-05-29 19:55:01,290-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1e8df2d] FINISH, GetHardwareInfoVDSCommand, log id: 45ed8dc4
>2018-05-29 19:55:01,291-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [208a7e25] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 19:55:01,292-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [208a7e25] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 19:55:01,303-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [1b52fefa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:55:01,311-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [7b954f46] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:55:01,316-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [73a5b513] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 19:55:01,327-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [73a5b513] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 50113953
>2018-05-29 19:55:01,703-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [73a5b513] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 19:55:01,703-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [73a5b513] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 50113953
>2018-05-29 19:55:01,707-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [73a5b513] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:55:01,708-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [73a5b513] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:55:01,711-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [73a5b513] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:55:01,712-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [73a5b513] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:55:01,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [73a5b513] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 19:55:01,716-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [73a5b513] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 19:55:01,719-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [73a5b513] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 19:55:01,722-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [73a5b513] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 73a5b513, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 19:55:01,728-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1c74aeac
>2018-05-29 19:55:01,728-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 19:55:01,827-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:55:01,827-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 19:55:01,827-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 19:55:01,827-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 19:55:01,827-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1c74aeac
>2018-05-29 19:55:01,827-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 19:55:01,841-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [21b70324] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:55:01,842-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [21b70324] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6157b6a4
>2018-05-29 19:55:01,844-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [21b70324] FINISH, SetVdsStatusVDSCommand, log id: 6157b6a4
>2018-05-29 19:55:01,876-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [21b70324] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 21b70324, Job ID: 69c91fce-efc8-4edd-95b5-07718058fe6e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 19:55:01,880-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [21b70324] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7b954f46, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 19:55:01,883-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [21b70324] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7b954f46, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 19:55:01,893-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [1218b081] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 19:55:01,894-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1218b081] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [23938283] Autorecovering 2 hosts
>2018-05-29 20:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [23938283] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:00:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [536fd64d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:00:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [536fd64d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:00:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [536fd64d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:00:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [536fd64d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:00:00,056-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [536fd64d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 655b16fd
>2018-05-29 20:00:00,058-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [536fd64d] FINISH, SetVdsStatusVDSCommand, log id: 655b16fd
>2018-05-29 20:00:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [536fd64d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:00:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [536fd64d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:00:00,062-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [536fd64d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:00:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [750a168b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:00:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [750a168b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:00:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [750a168b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:00:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [750a168b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:00:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [750a168b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3a971265
>2018-05-29 20:00:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [750a168b] FINISH, SetVdsStatusVDSCommand, log id: 3a971265
>2018-05-29 20:00:00,080-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [750a168b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 12c513e
>2018-05-29 20:00:00,083-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [750a168b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 12c513e
>2018-05-29 20:00:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [750a168b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:00:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [750a168b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:00:02,215-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [60599506] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 400e6032
>2018-05-29 20:00:02,216-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [60599506] FINISH, GetHardwareInfoVDSCommand, log id: 400e6032
>2018-05-29 20:00:02,229-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [64e2980e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:00:02,238-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [192cc2fb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:00:02,243-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [74a4d7d6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:00:02,250-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [74a4d7d6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e67ad11
>2018-05-29 20:00:02,585-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4e8ba6ad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4d2f64e3
>2018-05-29 20:00:02,586-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4e8ba6ad] FINISH, GetHardwareInfoVDSCommand, log id: 4d2f64e3
>2018-05-29 20:00:02,599-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [11ebdd82] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:00:02,603-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [7d0377b9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:00:02,608-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5cddcb08] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:00:02,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5cddcb08] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 30e096b7
>2018-05-29 20:00:02,626-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74a4d7d6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:00:02,626-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [74a4d7d6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e67ad11
>2018-05-29 20:00:02,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74a4d7d6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:00:02,631-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [74a4d7d6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:00:02,635-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74a4d7d6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:00:02,636-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [74a4d7d6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:00:02,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74a4d7d6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:00:02,640-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [74a4d7d6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:00:02,643-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [74a4d7d6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:00:02,645-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74a4d7d6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 74a4d7d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:00:02,648-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 11ed90a3
>2018-05-29 20:00:02,648-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:00:02,742-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:00:02,742-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:00:02,742-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:00:02,742-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:00:02,742-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 11ed90a3
>2018-05-29 20:00:02,742-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:00:02,755-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [551ebb2e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:00:02,756-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [551ebb2e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45aabb70
>2018-05-29 20:00:02,758-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [551ebb2e] FINISH, SetVdsStatusVDSCommand, log id: 45aabb70
>2018-05-29 20:00:02,788-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [551ebb2e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 551ebb2e, Job ID: e3196d3d-8766-4c40-8ae0-0ce5c364fc55, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:00:02,793-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [551ebb2e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 192cc2fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:00:02,795-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [551ebb2e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 192cc2fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:00:02,806-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [1e9e2cda] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:00:02,807-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1e9e2cda] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:00:03,978-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5cddcb08] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:00:03,978-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5cddcb08] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 30e096b7
>2018-05-29 20:00:03,981-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5cddcb08] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:00:03,982-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5cddcb08] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:00:03,986-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5cddcb08] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:00:03,986-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5cddcb08] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:00:03,990-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5cddcb08] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:00:03,991-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5cddcb08] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:00:03,994-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5cddcb08] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:00:03,996-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5cddcb08] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5cddcb08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:00:03,999-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 928cb12
>2018-05-29 20:00:03,999-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:00:04,096-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:00:04,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:00:04,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:00:04,096-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:00:04,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 928cb12
>2018-05-29 20:00:04,097-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:00:04,109-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [2871c67b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:00:04,110-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2871c67b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 288ceff2
>2018-05-29 20:00:04,112-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2871c67b] FINISH, SetVdsStatusVDSCommand, log id: 288ceff2
>2018-05-29 20:00:04,118-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2871c67b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2871c67b, Job ID: a969651f-8428-449d-b40f-fe4aad1fa5a8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:00:04,123-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2871c67b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7d0377b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:00:04,125-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2871c67b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7d0377b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:00:04,135-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [64b5e0fc] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:00:04,136-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [64b5e0fc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [6451bf01] Autorecovering 2 hosts
>2018-05-29 20:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [6451bf01] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:05:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7df165c4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:05:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7df165c4] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:05:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7df165c4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:05:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7df165c4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:05:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7df165c4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: bbd2bcb
>2018-05-29 20:05:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7df165c4] FINISH, SetVdsStatusVDSCommand, log id: bbd2bcb
>2018-05-29 20:05:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7df165c4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:05:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7df165c4] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:05:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7df165c4] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:05:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30f6e12a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:05:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30f6e12a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:05:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30f6e12a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:05:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30f6e12a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:05:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [30f6e12a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13ffbd48
>2018-05-29 20:05:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [30f6e12a] FINISH, SetVdsStatusVDSCommand, log id: 13ffbd48
>2018-05-29 20:05:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [30f6e12a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7596a29b
>2018-05-29 20:05:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [30f6e12a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7596a29b
>2018-05-29 20:05:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30f6e12a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:05:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30f6e12a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:05:00,417-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4b062408] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3672ce9a
>2018-05-29 20:05:00,419-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4b062408] FINISH, GetHardwareInfoVDSCommand, log id: 3672ce9a
>2018-05-29 20:05:00,431-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [2751f760] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:05:00,440-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [60f016e7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:05:00,445-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [648c9641] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:05:00,456-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [648c9641] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1ab50a99
>2018-05-29 20:05:00,839-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [648c9641] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:05:00,839-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [648c9641] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1ab50a99
>2018-05-29 20:05:00,843-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [648c9641] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:05:00,844-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [648c9641] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:05:00,848-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [648c9641] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:05:00,849-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [648c9641] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:05:00,852-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [648c9641] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:05:00,853-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [648c9641] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:05:00,856-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [648c9641] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:05:00,859-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [648c9641] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 648c9641, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:05:00,865-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 29948371
>2018-05-29 20:05:00,866-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:05:00,964-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:05:00,964-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:05:00,964-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:05:00,964-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:05:00,964-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 29948371
>2018-05-29 20:05:00,964-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:05:00,978-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [13b3c988] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:05:00,979-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [13b3c988] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39742eda
>2018-05-29 20:05:00,981-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [13b3c988] FINISH, SetVdsStatusVDSCommand, log id: 39742eda
>2018-05-29 20:05:01,012-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13b3c988] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 13b3c988, Job ID: 2883793e-5591-4f92-85c0-4b37907d0600, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:05:01,017-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13b3c988] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 60f016e7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:05:01,019-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13b3c988] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 60f016e7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:05:01,029-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [77fd19c4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:05:01,030-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [77fd19c4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:05:01,796-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6294a8bf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 461014e9
>2018-05-29 20:05:01,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6294a8bf] FINISH, GetHardwareInfoVDSCommand, log id: 461014e9
>2018-05-29 20:05:01,810-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2d7a0cf9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:05:01,815-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5af3eeb8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:05:01,819-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [62d9ef48] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:05:01,826-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [62d9ef48] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3af93a06
>2018-05-29 20:05:02,191-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62d9ef48] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:05:02,191-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [62d9ef48] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3af93a06
>2018-05-29 20:05:02,195-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62d9ef48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:05:02,195-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [62d9ef48] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:05:02,199-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62d9ef48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:05:02,200-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [62d9ef48] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:05:02,204-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62d9ef48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:05:02,205-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [62d9ef48] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:05:02,207-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [62d9ef48] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:05:02,210-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62d9ef48] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 62d9ef48, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:05:02,212-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 14db2b1f
>2018-05-29 20:05:02,212-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:05:02,307-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:05:02,307-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:05:02,307-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:05:02,307-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:05:02,307-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 14db2b1f
>2018-05-29 20:05:02,307-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:05:02,320-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [6bcd64fd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:05:02,321-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6bcd64fd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4c03a6cb
>2018-05-29 20:05:02,323-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6bcd64fd] FINISH, SetVdsStatusVDSCommand, log id: 4c03a6cb
>2018-05-29 20:05:02,329-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bcd64fd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6bcd64fd, Job ID: 644a1957-7dd3-44b3-b88b-ea078a6556b5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:05:02,333-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bcd64fd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5af3eeb8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:05:02,335-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bcd64fd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5af3eeb8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:05:02,345-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [32cf816] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:05:02,346-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [32cf816] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [23eada57] Autorecovering 2 hosts
>2018-05-29 20:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [23eada57] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:10:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a7ff7e3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a7ff7e3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a7ff7e3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a7ff7e3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:10:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6a7ff7e3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4db9a219
>2018-05-29 20:10:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6a7ff7e3] FINISH, SetVdsStatusVDSCommand, log id: 4db9a219
>2018-05-29 20:10:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a7ff7e3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:10:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a7ff7e3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:10:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6a7ff7e3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:10:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76c9b275] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:10:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76c9b275] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:10:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76c9b275] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:10:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76c9b275] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:10:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [76c9b275] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1e8803cc
>2018-05-29 20:10:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [76c9b275] FINISH, SetVdsStatusVDSCommand, log id: 1e8803cc
>2018-05-29 20:10:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [76c9b275] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 77787967
>2018-05-29 20:10:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [76c9b275] FINISH, SetHaMaintenanceModeVDSCommand, log id: 77787967
>2018-05-29 20:10:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76c9b275] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:10:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76c9b275] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:10:01,796-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2670160f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: b18545e
>2018-05-29 20:10:01,797-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2670160f] FINISH, GetHardwareInfoVDSCommand, log id: b18545e
>2018-05-29 20:10:01,810-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [40163ec7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:10:01,818-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [448622dd] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:10:01,823-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [13ed266c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:10:01,834-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [13ed266c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5f1b064e
>2018-05-29 20:10:02,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [13ed266c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:10:02,218-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [13ed266c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5f1b064e
>2018-05-29 20:10:02,222-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [13ed266c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:10:02,222-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [13ed266c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:10:02,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [13ed266c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:10:02,227-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [13ed266c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:10:02,231-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [13ed266c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:10:02,231-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [13ed266c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:10:02,234-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [13ed266c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:10:02,237-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [13ed266c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 13ed266c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:10:02,244-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 69fe9b38
>2018-05-29 20:10:02,244-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:10:02,341-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:10:02,341-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:10:02,341-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:10:02,341-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:10:02,341-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 69fe9b38
>2018-05-29 20:10:02,341-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:10:02,354-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [42ba87c7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:10:02,355-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [42ba87c7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 22fdb2e3
>2018-05-29 20:10:02,357-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [42ba87c7] FINISH, SetVdsStatusVDSCommand, log id: 22fdb2e3
>2018-05-29 20:10:02,388-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [42ba87c7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 42ba87c7, Job ID: 69a70be3-ff6f-4303-ad95-17f7b0b3527f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:10:02,393-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [42ba87c7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 448622dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:10:02,395-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [42ba87c7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 448622dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:10:02,406-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [436084c5] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:10:02,407-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [436084c5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:10:03,030-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [43934a74] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 36ed0985
>2018-05-29 20:10:03,032-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [43934a74] FINISH, GetHardwareInfoVDSCommand, log id: 36ed0985
>2018-05-29 20:10:03,045-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [3c820193] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:10:03,049-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [5bcf7856] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:10:03,054-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [d0df2d8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:10:03,061-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [d0df2d8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 16d0f4c
>2018-05-29 20:10:03,428-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d0df2d8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:10:03,428-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [d0df2d8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 16d0f4c
>2018-05-29 20:10:03,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d0df2d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:10:03,433-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [d0df2d8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:10:03,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d0df2d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:10:03,437-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [d0df2d8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:10:03,440-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d0df2d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:10:03,441-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [d0df2d8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:10:03,443-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [d0df2d8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:10:03,446-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d0df2d8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: d0df2d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:10:03,449-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3fdf8976
>2018-05-29 20:10:03,449-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:10:03,547-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:10:03,547-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:10:03,547-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:10:03,547-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:10:03,547-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3fdf8976
>2018-05-29 20:10:03,547-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:10:03,560-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [74ea1fd2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:10:03,561-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [74ea1fd2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 622c0557
>2018-05-29 20:10:03,563-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [74ea1fd2] FINISH, SetVdsStatusVDSCommand, log id: 622c0557
>2018-05-29 20:10:03,568-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74ea1fd2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 74ea1fd2, Job ID: 22810f62-180a-4008-9250-8f1f2f3b0788, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:10:03,573-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74ea1fd2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5bcf7856, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:10:03,575-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74ea1fd2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5bcf7856, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:10:03,586-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [61013dfc] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:10:03,587-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [61013dfc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:15:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [b3b774a] Autorecovering 2 hosts
>2018-05-29 20:15:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [b3b774a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:15:00,027-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d792420] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:15:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d792420] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:15:00,035-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d792420] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:15:00,035-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d792420] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:15:00,036-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7d792420] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4751c55b
>2018-05-29 20:15:00,039-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7d792420] FINISH, SetVdsStatusVDSCommand, log id: 4751c55b
>2018-05-29 20:15:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d792420] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:15:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d792420] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:15:00,041-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7d792420] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:15:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17fe2e68] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:15:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17fe2e68] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:15:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17fe2e68] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:15:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17fe2e68] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:15:00,055-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [17fe2e68] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c92e224
>2018-05-29 20:15:00,057-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [17fe2e68] FINISH, SetVdsStatusVDSCommand, log id: 7c92e224
>2018-05-29 20:15:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [17fe2e68] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 186e3774
>2018-05-29 20:15:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [17fe2e68] FINISH, SetHaMaintenanceModeVDSCommand, log id: 186e3774
>2018-05-29 20:15:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17fe2e68] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:15:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17fe2e68] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:15:01,221-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4580dfa0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7e895253
>2018-05-29 20:15:01,223-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4580dfa0] FINISH, GetHardwareInfoVDSCommand, log id: 7e895253
>2018-05-29 20:15:01,235-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [1c1eecfe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:15:01,240-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [4792a827] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:15:01,245-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5cd5a52a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:15:01,251-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5cd5a52a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 51d71992
>2018-05-29 20:15:01,624-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5cd5a52a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:15:01,624-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5cd5a52a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 51d71992
>2018-05-29 20:15:01,628-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5cd5a52a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:15:01,629-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5cd5a52a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:15:01,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5cd5a52a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:15:01,633-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5cd5a52a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:15:01,637-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5cd5a52a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:15:01,638-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5cd5a52a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:15:01,641-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5cd5a52a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:15:01,643-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5cd5a52a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5cd5a52a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:15:01,646-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4f4e31c1
>2018-05-29 20:15:01,646-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:15:01,745-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:15:01,745-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:15:01,745-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:15:01,745-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:15:01,745-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4f4e31c1
>2018-05-29 20:15:01,745-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:15:01,758-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [2b189751] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:15:01,759-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2b189751] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1dc73ef5
>2018-05-29 20:15:01,761-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2b189751] FINISH, SetVdsStatusVDSCommand, log id: 1dc73ef5
>2018-05-29 20:15:01,767-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2b189751] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2b189751, Job ID: 2fe445bc-6a3e-4169-b92a-26da9d7bc4bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:15:01,772-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2b189751] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4792a827, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:15:01,774-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2b189751] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4792a827, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:15:01,785-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [38936568] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:15:01,786-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [38936568] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:15:02,982-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [10808255] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 719c4b81
>2018-05-29 20:15:02,983-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [10808255] FINISH, GetHardwareInfoVDSCommand, log id: 719c4b81
>2018-05-29 20:15:02,996-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [384121e0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:15:03,000-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2665d96] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:15:03,005-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [463e43ed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:15:03,012-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [463e43ed] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7fba9616
>2018-05-29 20:15:03,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [463e43ed] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:15:03,389-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [463e43ed] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7fba9616
>2018-05-29 20:15:03,392-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [463e43ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:15:03,393-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [463e43ed] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:15:03,397-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [463e43ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:15:03,398-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [463e43ed] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:15:03,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [463e43ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:15:03,402-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [463e43ed] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:15:03,404-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [463e43ed] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:15:03,407-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [463e43ed] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 463e43ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:15:03,410-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 45c9e74d
>2018-05-29 20:15:03,410-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:15:03,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:15:03,508-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:15:03,508-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:15:03,508-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:15:03,508-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 45c9e74d
>2018-05-29 20:15:03,508-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:15:03,521-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [8f93b81] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:15:03,522-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [8f93b81] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 48fe4ed8
>2018-05-29 20:15:03,524-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [8f93b81] FINISH, SetVdsStatusVDSCommand, log id: 48fe4ed8
>2018-05-29 20:15:03,530-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8f93b81] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 8f93b81, Job ID: 7f14dfac-cd26-46cf-a9e0-49a02eb83894, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:15:03,534-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8f93b81] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2665d96, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:15:03,536-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8f93b81] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2665d96, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:15:03,547-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [34e3a8f0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:15:03,548-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [34e3a8f0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:20:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1fe1afb] Autorecovering 2 hosts
>2018-05-29 20:20:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1fe1afb] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:20:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2345b48c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:20:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2345b48c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:20:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2345b48c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:20:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2345b48c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:20:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2345b48c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 91bf2c5
>2018-05-29 20:20:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2345b48c] FINISH, SetVdsStatusVDSCommand, log id: 91bf2c5
>2018-05-29 20:20:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2345b48c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:20:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2345b48c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:20:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [2345b48c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:20:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [14b958e7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:20:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [14b958e7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:20:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [14b958e7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:20:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [14b958e7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:20:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [14b958e7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 592911e3
>2018-05-29 20:20:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [14b958e7] FINISH, SetVdsStatusVDSCommand, log id: 592911e3
>2018-05-29 20:20:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [14b958e7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7b8ddaee
>2018-05-29 20:20:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [14b958e7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7b8ddaee
>2018-05-29 20:20:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [14b958e7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:20:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [14b958e7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:20:01,172-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [38e2b4db] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 33c485ae
>2018-05-29 20:20:01,173-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [38e2b4db] FINISH, GetHardwareInfoVDSCommand, log id: 33c485ae
>2018-05-29 20:20:01,187-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [257fae78] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:20:01,195-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [5c8acf93] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:20:01,201-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3e34bf3c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:20:01,212-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3e34bf3c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 85b51c7
>2018-05-29 20:20:01,588-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e34bf3c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:20:01,588-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3e34bf3c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 85b51c7
>2018-05-29 20:20:01,592-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e34bf3c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:20:01,593-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3e34bf3c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:20:01,597-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e34bf3c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:20:01,597-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3e34bf3c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:20:01,601-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e34bf3c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:20:01,602-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3e34bf3c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:20:01,604-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3e34bf3c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:20:01,607-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e34bf3c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3e34bf3c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:20:01,613-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 62bb3cdc
>2018-05-29 20:20:01,613-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:20:01,711-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:20:01,711-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:20:01,711-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:20:01,711-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:20:01,711-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 62bb3cdc
>2018-05-29 20:20:01,711-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:20:01,724-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [22589a4e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:20:01,725-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [22589a4e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 28fded56
>2018-05-29 20:20:01,728-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [22589a4e] FINISH, SetVdsStatusVDSCommand, log id: 28fded56
>2018-05-29 20:20:01,759-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [22589a4e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22589a4e, Job ID: d8eeb5e6-22f2-41d7-b29d-af359c459205, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:20:01,764-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [22589a4e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5c8acf93, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:20:01,766-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [22589a4e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5c8acf93, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:20:01,776-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [21f92774] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:20:01,777-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [21f92774] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:20:02,471-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [157022f2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4419d57c
>2018-05-29 20:20:02,473-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [157022f2] FINISH, GetHardwareInfoVDSCommand, log id: 4419d57c
>2018-05-29 20:20:02,486-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [3934a822] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:20:02,491-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [5ca61db2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:20:02,496-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5a3db630] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:20:02,503-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5a3db630] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3a563acc
>2018-05-29 20:20:03,870-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a3db630] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:20:03,870-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5a3db630] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3a563acc
>2018-05-29 20:20:03,874-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a3db630] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:20:03,874-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5a3db630] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:20:03,878-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a3db630] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:20:03,879-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5a3db630] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:20:03,883-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a3db630] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:20:03,884-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5a3db630] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:20:03,887-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5a3db630] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:20:03,890-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a3db630] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5a3db630, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:20:03,893-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 52b8dbfb
>2018-05-29 20:20:03,893-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:20:03,990-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:20:03,990-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:20:03,990-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:20:03,990-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:20:03,990-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 52b8dbfb
>2018-05-29 20:20:03,990-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:20:04,003-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [9207b4d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:20:04,004-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [9207b4d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2305101c
>2018-05-29 20:20:04,006-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [9207b4d] FINISH, SetVdsStatusVDSCommand, log id: 2305101c
>2018-05-29 20:20:04,012-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [9207b4d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 9207b4d, Job ID: 622300af-9748-480f-b3ae-533b1bb7a540, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:20:04,016-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [9207b4d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5ca61db2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:20:04,018-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [9207b4d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5ca61db2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:20:04,029-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [1a0ca76f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:20:04,030-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [1a0ca76f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [41f39184] Autorecovering 2 hosts
>2018-05-29 20:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [41f39184] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:25:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1b7b6339] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:25:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1b7b6339] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:25:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1b7b6339] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:25:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1b7b6339] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:25:00,050-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1b7b6339] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f545348
>2018-05-29 20:25:00,053-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1b7b6339] FINISH, SetVdsStatusVDSCommand, log id: 1f545348
>2018-05-29 20:25:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1b7b6339] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:25:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1b7b6339] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:25:00,055-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1b7b6339] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:25:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3afd8230] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:25:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3afd8230] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:25:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3afd8230] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:25:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3afd8230] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:25:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3afd8230] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 503d8dc6
>2018-05-29 20:25:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3afd8230] FINISH, SetVdsStatusVDSCommand, log id: 503d8dc6
>2018-05-29 20:25:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [3afd8230] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2ede1ddd
>2018-05-29 20:25:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [3afd8230] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2ede1ddd
>2018-05-29 20:25:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3afd8230] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:25:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3afd8230] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:25:01,663-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5a6e39bf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 44923ea2
>2018-05-29 20:25:01,665-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5a6e39bf] FINISH, GetHardwareInfoVDSCommand, log id: 44923ea2
>2018-05-29 20:25:01,678-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [647b4476] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:25:01,687-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [7cdaa4cc] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:25:01,693-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [4773c29] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:25:01,706-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [4773c29] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6c08112a
>2018-05-29 20:25:02,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4773c29] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:25:02,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [4773c29] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6c08112a
>2018-05-29 20:25:02,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4773c29] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:25:02,082-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [4773c29] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:25:02,086-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4773c29] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:25:02,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [4773c29] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:25:02,090-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4773c29] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:25:02,091-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [4773c29] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:25:02,094-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [4773c29] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:25:02,097-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4773c29] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4773c29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:25:02,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7546c712
>2018-05-29 20:25:02,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:25:02,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:25:02,198-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:25:02,198-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:25:02,198-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:25:02,198-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7546c712
>2018-05-29 20:25:02,198-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:25:02,211-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [33a90158] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:25:02,212-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [33a90158] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 69a35145
>2018-05-29 20:25:02,214-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [33a90158] FINISH, SetVdsStatusVDSCommand, log id: 69a35145
>2018-05-29 20:25:02,244-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [33a90158] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 33a90158, Job ID: ce552a05-bf82-49dd-8657-9a0b791839d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:25:02,248-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [33a90158] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7cdaa4cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:25:02,250-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [33a90158] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7cdaa4cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:25:02,261-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [a1d6446] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:25:02,262-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [a1d6446] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:25:02,399-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7745d6f4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 111c6d94
>2018-05-29 20:25:02,400-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7745d6f4] FINISH, GetHardwareInfoVDSCommand, log id: 111c6d94
>2018-05-29 20:25:02,412-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4aa8989] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:25:02,421-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [1a2e901] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:25:02,426-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [477691b2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:25:02,432-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [477691b2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 624d5e8f
>2018-05-29 20:25:02,805-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [477691b2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:25:02,805-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [477691b2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 624d5e8f
>2018-05-29 20:25:02,809-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [477691b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:25:02,809-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [477691b2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:25:02,813-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [477691b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:25:02,814-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [477691b2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:25:02,818-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [477691b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:25:02,818-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [477691b2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:25:02,821-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [477691b2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:25:02,823-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [477691b2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 477691b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:25:02,826-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3753487c
>2018-05-29 20:25:02,826-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:25:02,920-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:25:02,921-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:25:02,921-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:25:02,921-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:25:02,921-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3753487c
>2018-05-29 20:25:02,921-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:25:02,933-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [65ee40f1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:25:02,935-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [65ee40f1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 67fcae6b
>2018-05-29 20:25:02,937-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [65ee40f1] FINISH, SetVdsStatusVDSCommand, log id: 67fcae6b
>2018-05-29 20:25:02,943-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [65ee40f1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 65ee40f1, Job ID: 2895d503-4cba-4fac-9a45-f07345d3f1cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:25:02,947-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [65ee40f1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1a2e901, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:25:02,949-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [65ee40f1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1a2e901, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:25:02,959-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [36f2d41] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:25:02,960-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [36f2d41] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [2af9e0d6] Autorecovering 2 hosts
>2018-05-29 20:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [2af9e0d6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:30:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [efb5c36] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:30:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [efb5c36] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:30:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [efb5c36] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:30:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [efb5c36] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:30:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [efb5c36] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6101a4c5
>2018-05-29 20:30:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [efb5c36] FINISH, SetVdsStatusVDSCommand, log id: 6101a4c5
>2018-05-29 20:30:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [efb5c36] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:30:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [efb5c36] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:30:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [efb5c36] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:30:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [17b4abe7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:30:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [17b4abe7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:30:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [17b4abe7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:30:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [17b4abe7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:30:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [17b4abe7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76e4f338
>2018-05-29 20:30:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [17b4abe7] FINISH, SetVdsStatusVDSCommand, log id: 76e4f338
>2018-05-29 20:30:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [17b4abe7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3ef72963
>2018-05-29 20:30:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [17b4abe7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3ef72963
>2018-05-29 20:30:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [17b4abe7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:30:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [17b4abe7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:30:00,553-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [405d5f2d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7a93d7fe
>2018-05-29 20:30:00,555-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [405d5f2d] FINISH, GetHardwareInfoVDSCommand, log id: 7a93d7fe
>2018-05-29 20:30:00,568-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [6d03b85b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:30:00,573-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [15e586d9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:30:00,578-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3d968777] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:30:00,591-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3d968777] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6e6170fa
>2018-05-29 20:30:00,974-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d968777] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:30:00,974-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3d968777] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6e6170fa
>2018-05-29 20:30:00,978-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d968777] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:30:00,979-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3d968777] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:30:00,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d968777] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:30:00,983-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3d968777] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:30:00,988-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d968777] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:30:00,988-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3d968777] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:30:00,991-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3d968777] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:30:00,994-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d968777] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3d968777, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:30:01,000-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3202a2a0
>2018-05-29 20:30:01,000-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:30:01,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:30:01,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:30:01,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:30:01,099-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:30:01,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3202a2a0
>2018-05-29 20:30:01,099-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:30:01,112-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [770eaa99] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:30:01,114-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [770eaa99] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5420d97b
>2018-05-29 20:30:01,116-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [770eaa99] FINISH, SetVdsStatusVDSCommand, log id: 5420d97b
>2018-05-29 20:30:01,147-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [770eaa99] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 770eaa99, Job ID: a7f756d0-e6c7-4604-b372-85c7df083017, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:30:01,152-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [770eaa99] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 15e586d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:30:01,154-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [770eaa99] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 15e586d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:30:01,165-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [65c75cbb] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:30:01,166-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [65c75cbb] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:30:02,858-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [3eb840a1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 597af2f0
>2018-05-29 20:30:02,860-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [3eb840a1] FINISH, GetHardwareInfoVDSCommand, log id: 597af2f0
>2018-05-29 20:30:02,873-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [10bb7ba6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:30:02,878-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [1f79c1ec] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:30:02,883-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [75fa290e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:30:02,889-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [75fa290e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 24deab5e
>2018-05-29 20:30:04,252-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75fa290e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:30:04,252-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [75fa290e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 24deab5e
>2018-05-29 20:30:04,256-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75fa290e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:30:04,257-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [75fa290e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:30:04,260-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75fa290e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:30:04,261-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [75fa290e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:30:04,265-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75fa290e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:30:04,265-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [75fa290e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:30:04,268-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [75fa290e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:30:04,271-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75fa290e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 75fa290e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:30:04,274-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6337d758
>2018-05-29 20:30:04,274-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:30:04,373-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:30:04,373-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:30:04,373-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:30:04,373-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:30:04,373-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6337d758
>2018-05-29 20:30:04,373-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:30:04,386-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [8f6c2aa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:30:04,387-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [8f6c2aa] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 66d841f6
>2018-05-29 20:30:04,389-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [8f6c2aa] FINISH, SetVdsStatusVDSCommand, log id: 66d841f6
>2018-05-29 20:30:04,395-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [8f6c2aa] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 8f6c2aa, Job ID: d2acdbcf-940c-4d36-82cf-3af70398a0e1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:30:04,399-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [8f6c2aa] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1f79c1ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:30:04,401-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [8f6c2aa] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1f79c1ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:30:04,411-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [1e3c563] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:30:04,412-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [1e3c563] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [719b2a49] Autorecovering 2 hosts
>2018-05-29 20:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [719b2a49] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:35:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9f31913] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:35:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9f31913] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:35:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9f31913] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:35:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9f31913] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:35:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [9f31913] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5738975c
>2018-05-29 20:35:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [9f31913] FINISH, SetVdsStatusVDSCommand, log id: 5738975c
>2018-05-29 20:35:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9f31913] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:35:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9f31913] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:35:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [9f31913] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:35:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2a621bc2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:35:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2a621bc2] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:35:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2a621bc2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:35:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2a621bc2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:35:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2a621bc2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5b169133
>2018-05-29 20:35:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2a621bc2] FINISH, SetVdsStatusVDSCommand, log id: 5b169133
>2018-05-29 20:35:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [2a621bc2] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1d10078
>2018-05-29 20:35:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [2a621bc2] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1d10078
>2018-05-29 20:35:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2a621bc2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:35:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2a621bc2] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:35:01,941-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [59226faa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2f5eafe6
>2018-05-29 20:35:01,943-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [59226faa] FINISH, GetHardwareInfoVDSCommand, log id: 2f5eafe6
>2018-05-29 20:35:01,955-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2fa5c669] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:35:01,964-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [52415143] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:35:01,969-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [a7179dc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:35:01,981-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [a7179dc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1fc442e1
>2018-05-29 20:35:02,017-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1def6e68] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1b36b63
>2018-05-29 20:35:02,019-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1def6e68] FINISH, GetHardwareInfoVDSCommand, log id: 1b36b63
>2018-05-29 20:35:02,031-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7ac92934] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:35:02,036-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [7199b880] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:35:02,041-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4fd1ab93] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:35:02,047-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4fd1ab93] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3ca5b1cd
>2018-05-29 20:35:02,366-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [a7179dc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:35:02,366-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [a7179dc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1fc442e1
>2018-05-29 20:35:02,370-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [a7179dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:35:02,371-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [a7179dc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:35:02,375-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [a7179dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:35:02,375-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [a7179dc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:35:02,379-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [a7179dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:35:02,380-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [a7179dc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:35:02,383-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [a7179dc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:35:02,385-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [a7179dc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: a7179dc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:35:02,392-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 41a20c9e
>2018-05-29 20:35:02,392-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:35:02,418-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4fd1ab93] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:35:02,418-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4fd1ab93] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3ca5b1cd
>2018-05-29 20:35:02,422-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4fd1ab93] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:35:02,422-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4fd1ab93] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:35:02,426-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4fd1ab93] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:35:02,427-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4fd1ab93] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:35:02,430-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4fd1ab93] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:35:02,431-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4fd1ab93] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:35:02,434-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4fd1ab93] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:35:02,436-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4fd1ab93] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4fd1ab93, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:35:02,489-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:35:02,489-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:35:02,489-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:35:02,489-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:35:02,489-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 41a20c9e
>2018-05-29 20:35:02,489-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:35:02,492-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7039f10d
>2018-05-29 20:35:02,492-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:35:02,504-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [5297d7b8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:35:02,505-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5297d7b8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 21e4cb22
>2018-05-29 20:35:02,507-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5297d7b8] FINISH, SetVdsStatusVDSCommand, log id: 21e4cb22
>2018-05-29 20:35:02,538-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5297d7b8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5297d7b8, Job ID: 8e554deb-8ee1-4186-9c88-2026aa630433, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:35:02,543-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5297d7b8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 52415143, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:35:02,545-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5297d7b8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 52415143, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:35:02,556-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [35a1e8c3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:35:02,557-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [35a1e8c3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:35:02,591-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:35:02,591-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:35:02,591-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:35:02,591-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:35:02,591-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7039f10d
>2018-05-29 20:35:02,591-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:35:02,603-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [49340430] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:35:02,605-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [49340430] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 727a368
>2018-05-29 20:35:02,607-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [49340430] FINISH, SetVdsStatusVDSCommand, log id: 727a368
>2018-05-29 20:35:02,612-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [49340430] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 49340430, Job ID: c897426c-8cac-407f-bada-f51014a9d270, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:35:02,616-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [49340430] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7199b880, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:35:02,619-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [49340430] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7199b880, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:35:02,629-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [6a2163b7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:35:02,630-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [6a2163b7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [67613628] Autorecovering 2 hosts
>2018-05-29 20:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [67613628] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:40:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1966e3b1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1966e3b1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1966e3b1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1966e3b1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:40:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1966e3b1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 597befb6
>2018-05-29 20:40:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1966e3b1] FINISH, SetVdsStatusVDSCommand, log id: 597befb6
>2018-05-29 20:40:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1966e3b1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:40:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1966e3b1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:40:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [1966e3b1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:40:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4fb61436] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:40:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4fb61436] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:40:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4fb61436] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:40:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4fb61436] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:40:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4fb61436] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 28d7d784
>2018-05-29 20:40:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4fb61436] FINISH, SetVdsStatusVDSCommand, log id: 28d7d784
>2018-05-29 20:40:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4fb61436] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 50b1ac97
>2018-05-29 20:40:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4fb61436] FINISH, SetHaMaintenanceModeVDSCommand, log id: 50b1ac97
>2018-05-29 20:40:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4fb61436] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:40:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4fb61436] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:40:00,215-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [497f18e2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1f79c083
>2018-05-29 20:40:00,216-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [497f18e2] FINISH, GetHardwareInfoVDSCommand, log id: 1f79c083
>2018-05-29 20:40:00,229-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [1d3239c6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:40:00,238-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [211cca15] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:40:00,240-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4a4e4896] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 8afc064
>2018-05-29 20:40:00,242-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4a4e4896] FINISH, GetHardwareInfoVDSCommand, log id: 8afc064
>2018-05-29 20:40:00,244-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2d2cd9e0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:40:00,251-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2d2cd9e0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4b21e393
>2018-05-29 20:40:00,255-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [53ff17c2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:40:00,260-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [18ad2692] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:40:00,265-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [52c79308] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:40:00,278-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [52c79308] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4d48ec74
>2018-05-29 20:40:00,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2d2cd9e0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:40:00,631-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2d2cd9e0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4b21e393
>2018-05-29 20:40:00,635-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2d2cd9e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:40:00,636-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2d2cd9e0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:40:00,640-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2d2cd9e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:40:00,641-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2d2cd9e0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:40:00,645-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52c79308] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:40:00,645-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [52c79308] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4d48ec74
>2018-05-29 20:40:00,646-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2d2cd9e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:40:00,646-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2d2cd9e0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:40:00,649-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52c79308] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:40:00,649-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [52c79308] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:40:00,650-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2d2cd9e0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:40:00,653-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2d2cd9e0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2d2cd9e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:40:00,654-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52c79308] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:40:00,655-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [52c79308] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:40:00,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52c79308] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:40:00,659-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [52c79308] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:40:00,660-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2daa4d45
>2018-05-29 20:40:00,660-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:40:00,662-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [52c79308] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:40:00,665-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52c79308] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 52c79308, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:40:00,755-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:40:00,755-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:40:00,755-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:40:00,755-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:40:00,755-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2daa4d45
>2018-05-29 20:40:00,755-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:40:00,756-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 64b75b86
>2018-05-29 20:40:00,756-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:40:00,768-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [57807fac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:40:00,770-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [57807fac] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 27116732
>2018-05-29 20:40:00,772-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [57807fac] FINISH, SetVdsStatusVDSCommand, log id: 27116732
>2018-05-29 20:40:00,805-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [57807fac] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 57807fac, Job ID: a9afe43d-6122-44b5-b2e2-d7be39f54165, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:40:00,809-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [57807fac] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 211cca15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:40:00,811-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [57807fac] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 211cca15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:40:00,822-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [19515bd] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:40:00,823-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [19515bd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:40:00,854-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:40:00,854-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:40:00,854-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:40:00,854-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:40:00,854-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 64b75b86
>2018-05-29 20:40:00,854-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:40:00,867-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4f62f049] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:40:00,869-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4f62f049] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 424473d9
>2018-05-29 20:40:00,871-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4f62f049] FINISH, SetVdsStatusVDSCommand, log id: 424473d9
>2018-05-29 20:40:00,876-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f62f049] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4f62f049, Job ID: 6decfd96-33d1-4b40-af44-91b6ebb4641f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:40:00,881-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f62f049] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 18ad2692, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:40:00,883-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f62f049] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 18ad2692, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:40:00,893-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [2a387ba6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:40:00,894-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2a387ba6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1c367d93] Autorecovering 2 hosts
>2018-05-29 20:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1c367d93] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:45:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7d064d62] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7d064d62] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7d064d62] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7d064d62] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:45:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7d064d62] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: c5ad69c
>2018-05-29 20:45:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7d064d62] FINISH, SetVdsStatusVDSCommand, log id: c5ad69c
>2018-05-29 20:45:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7d064d62] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:45:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7d064d62] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:45:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7d064d62] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:45:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [40901261] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:45:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [40901261] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:45:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [40901261] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:45:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [40901261] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:45:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [40901261] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: e605e2a
>2018-05-29 20:45:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [40901261] FINISH, SetVdsStatusVDSCommand, log id: e605e2a
>2018-05-29 20:45:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [40901261] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5ee1eafa
>2018-05-29 20:45:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [40901261] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5ee1eafa
>2018-05-29 20:45:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [40901261] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:45:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [40901261] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:45:01,739-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1a0db928] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 730b7b3
>2018-05-29 20:45:01,740-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1a0db928] FINISH, GetHardwareInfoVDSCommand, log id: 730b7b3
>2018-05-29 20:45:01,753-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [bbcea95] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:45:01,761-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [1f3f7342] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:45:01,767-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6b42a8e0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:45:01,779-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6b42a8e0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 525bb020
>2018-05-29 20:45:01,818-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [37e65e3d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6b4bce7c
>2018-05-29 20:45:01,820-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [37e65e3d] FINISH, GetHardwareInfoVDSCommand, log id: 6b4bce7c
>2018-05-29 20:45:01,833-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [42302273] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:45:01,837-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [71cf6ec0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:45:01,842-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [64888bbf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:45:01,849-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [64888bbf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1eb8c021
>2018-05-29 20:45:02,158-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6b42a8e0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:45:02,158-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6b42a8e0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 525bb020
>2018-05-29 20:45:02,162-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6b42a8e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:45:02,163-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6b42a8e0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:45:02,167-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6b42a8e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:45:02,167-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6b42a8e0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:45:02,171-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6b42a8e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:45:02,172-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6b42a8e0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:45:02,174-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6b42a8e0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:45:02,177-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6b42a8e0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6b42a8e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:45:02,183-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4121745e
>2018-05-29 20:45:02,184-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:45:02,222-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [64888bbf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:45:02,222-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [64888bbf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1eb8c021
>2018-05-29 20:45:02,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [64888bbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:45:02,226-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [64888bbf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:45:02,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [64888bbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:45:02,231-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [64888bbf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:45:02,234-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [64888bbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:45:02,235-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [64888bbf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:45:02,238-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [64888bbf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:45:02,240-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [64888bbf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 64888bbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:45:02,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:45:02,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:45:02,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:45:02,280-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:45:02,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4121745e
>2018-05-29 20:45:02,280-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:45:02,282-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a82df60
>2018-05-29 20:45:02,282-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:45:02,294-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [163931e6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:45:02,295-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [163931e6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c846668
>2018-05-29 20:45:02,297-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [163931e6] FINISH, SetVdsStatusVDSCommand, log id: 7c846668
>2018-05-29 20:45:02,328-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [163931e6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 163931e6, Job ID: 5e37b18b-63c3-42ba-9fc2-77091a8ad185, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:45:02,332-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [163931e6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1f3f7342, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:45:02,334-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [163931e6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1f3f7342, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:45:02,345-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [6e3b1b06] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:45:02,346-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [6e3b1b06] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:45:02,383-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:45:02,383-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:45:02,383-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:45:02,383-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:45:02,383-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a82df60
>2018-05-29 20:45:02,383-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:45:02,396-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [50037b06] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:45:02,397-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [50037b06] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 24dd708c
>2018-05-29 20:45:02,400-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [50037b06] FINISH, SetVdsStatusVDSCommand, log id: 24dd708c
>2018-05-29 20:45:02,433-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [50037b06] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 50037b06, Job ID: ec02d248-e42d-4809-ad5b-37aef69faf83, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:45:02,438-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [50037b06] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 71cf6ec0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:45:02,440-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [50037b06] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 71cf6ec0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:45:02,450-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [73905950] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:45:02,451-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [73905950] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7a3a692e] Autorecovering 2 hosts
>2018-05-29 20:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7a3a692e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:50:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56801ecd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:50:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56801ecd] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:50:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56801ecd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:50:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56801ecd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:50:00,048-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [56801ecd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33c646dc
>2018-05-29 20:50:00,050-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [56801ecd] FINISH, SetVdsStatusVDSCommand, log id: 33c646dc
>2018-05-29 20:50:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56801ecd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:50:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56801ecd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:50:00,053-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [56801ecd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:50:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f7457c3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:50:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f7457c3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:50:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f7457c3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:50:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f7457c3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:50:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [f7457c3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2fc33926
>2018-05-29 20:50:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [f7457c3] FINISH, SetVdsStatusVDSCommand, log id: 2fc33926
>2018-05-29 20:50:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [f7457c3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: a1cf03e
>2018-05-29 20:50:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [f7457c3] FINISH, SetHaMaintenanceModeVDSCommand, log id: a1cf03e
>2018-05-29 20:50:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f7457c3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:50:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f7457c3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:50:03,071-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7c989ecf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5c47ae64
>2018-05-29 20:50:03,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7c989ecf] FINISH, GetHardwareInfoVDSCommand, log id: 5c47ae64
>2018-05-29 20:50:03,080-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [586f18d6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7579b403
>2018-05-29 20:50:03,081-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [586f18d6] FINISH, GetHardwareInfoVDSCommand, log id: 7579b403
>2018-05-29 20:50:03,087-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4b0e2bdd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:50:03,094-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [362e12f9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:50:03,098-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [3dacae4] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:50:03,099-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [59e2c72a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:50:03,103-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [650f9c4b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:50:03,104-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [26e222c5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:50:03,116-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [650f9c4b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 35efa60d
>2018-05-29 20:50:03,117-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [26e222c5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3730a043
>2018-05-29 20:50:03,501-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26e222c5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:50:03,501-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [26e222c5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3730a043
>2018-05-29 20:50:03,502-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [650f9c4b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:50:03,502-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [650f9c4b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 35efa60d
>2018-05-29 20:50:03,505-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26e222c5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:50:03,506-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [26e222c5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:50:03,506-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [650f9c4b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:50:03,507-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [650f9c4b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:50:03,510-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26e222c5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:50:03,510-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [26e222c5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:50:03,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [650f9c4b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:50:03,512-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [650f9c4b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:50:03,514-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26e222c5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:50:03,515-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [26e222c5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:50:03,518-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [26e222c5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:50:03,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [650f9c4b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:50:03,520-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [650f9c4b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:50:03,524-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [650f9c4b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:50:03,524-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26e222c5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 26e222c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:50:03,526-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [650f9c4b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 650f9c4b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:50:03,531-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7fdd4d64
>2018-05-29 20:50:03,531-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:50:03,624-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:50:03,624-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:50:03,624-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:50:03,624-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:50:03,624-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7fdd4d64
>2018-05-29 20:50:03,624-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:50:03,626-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1623ce03
>2018-05-29 20:50:03,626-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:50:03,637-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2dfbd79] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:50:03,639-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2dfbd79] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ec61568
>2018-05-29 20:50:03,641-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2dfbd79] FINISH, SetVdsStatusVDSCommand, log id: 2ec61568
>2018-05-29 20:50:03,672-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2dfbd79] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2dfbd79, Job ID: 3b62b387-52fa-4334-8973-4a7e23b3f036, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:50:03,677-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2dfbd79] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 59e2c72a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:50:03,679-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2dfbd79] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 59e2c72a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:50:03,689-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [75474944] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:50:03,690-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [75474944] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:50:03,727-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:50:03,727-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:50:03,727-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:50:03,727-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:50:03,727-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1623ce03
>2018-05-29 20:50:03,727-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:50:03,740-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [103dc3c2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:50:03,741-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [103dc3c2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1bb6cb01
>2018-05-29 20:50:03,744-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [103dc3c2] FINISH, SetVdsStatusVDSCommand, log id: 1bb6cb01
>2018-05-29 20:50:03,749-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [103dc3c2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 103dc3c2, Job ID: c0dc0b1e-bdae-4a60-9ca0-1ef4aea528d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:50:03,754-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [103dc3c2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3dacae4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:50:03,756-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [103dc3c2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3dacae4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:50:03,766-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [56737be9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:50:03,767-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [56737be9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:54:43,266-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler7) [4cef5a96] Attempting to update VMs/Templates Ovf.
>2018-05-29 20:54:43,269-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 20:54:43,269-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 20:54:43,270-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:54:43,317-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 20:54:43,321-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 20:54:43,321-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 20:54:43,321-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 20:54:43,321-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 20:54:43,322-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 20:54:43,322-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 20:55:00,014-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [57505fce] Autorecovering 2 hosts
>2018-05-29 20:55:00,014-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [57505fce] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:55:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [43d49d04] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:55:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [43d49d04] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:55:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [43d49d04] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:55:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [43d49d04] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:55:00,048-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [43d49d04] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61a84295
>2018-05-29 20:55:00,051-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [43d49d04] FINISH, SetVdsStatusVDSCommand, log id: 61a84295
>2018-05-29 20:55:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [43d49d04] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:55:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [43d49d04] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 20:55:00,054-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [43d49d04] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:55:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1f5d797e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:55:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1f5d797e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 20:55:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1f5d797e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:55:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1f5d797e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:55:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1f5d797e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 711a508f
>2018-05-29 20:55:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1f5d797e] FINISH, SetVdsStatusVDSCommand, log id: 711a508f
>2018-05-29 20:55:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [1f5d797e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 23f79dc1
>2018-05-29 20:55:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [1f5d797e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 23f79dc1
>2018-05-29 20:55:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1f5d797e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 20:55:00,079-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1f5d797e] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 1f5d797e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-29 20:55:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1f5d797e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 20:55:01,354-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2ea234e9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2114735a
>2018-05-29 20:55:01,356-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2ea234e9] FINISH, GetHardwareInfoVDSCommand, log id: 2114735a
>2018-05-29 20:55:01,369-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [25e1b6bb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:55:01,378-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [58cd2626] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:55:01,381-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2badf97] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3e1c1140
>2018-05-29 20:55:01,382-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2badf97] FINISH, GetHardwareInfoVDSCommand, log id: 3e1c1140
>2018-05-29 20:55:01,383-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [721c18d8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:55:01,395-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [7c722009] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:55:01,397-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [721c18d8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 32f7ab5f
>2018-05-29 20:55:01,404-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [441071c5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:55:01,409-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [765fa309] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 20:55:01,421-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [765fa309] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 72acf2b5
>2018-05-29 20:55:01,779-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [721c18d8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:55:01,779-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [721c18d8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 32f7ab5f
>2018-05-29 20:55:01,783-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [721c18d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:55:01,784-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [721c18d8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:55:01,788-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [721c18d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:55:01,788-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [721c18d8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:55:01,790-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [765fa309] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 20:55:01,790-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [765fa309] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 72acf2b5
>2018-05-29 20:55:01,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [721c18d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:55:01,793-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [721c18d8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:55:01,794-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [765fa309] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:55:01,795-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [765fa309] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:55:01,797-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [721c18d8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:55:01,798-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [765fa309] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:55:01,799-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [765fa309] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:55:01,800-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [721c18d8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 721c18d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:55:01,803-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [765fa309] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 20:55:01,804-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [765fa309] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 20:55:01,806-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [765fa309] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 20:55:01,807-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 16e3919a
>2018-05-29 20:55:01,807-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:55:01,809-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [765fa309] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 765fa309, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 20:55:01,902-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:55:01,902-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:55:01,902-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:55:01,902-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:55:01,902-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 16e3919a
>2018-05-29 20:55:01,902-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:55:01,908-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 51e94b1
>2018-05-29 20:55:01,908-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 20:55:01,916-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [77cf89cb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:55:01,917-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [77cf89cb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19a139d3
>2018-05-29 20:55:01,919-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [77cf89cb] FINISH, SetVdsStatusVDSCommand, log id: 19a139d3
>2018-05-29 20:55:01,928-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77cf89cb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 77cf89cb, Job ID: f60237cd-dfbe-475f-acf0-089d9cd34ade, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:55:01,933-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77cf89cb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 58cd2626, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:55:01,935-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77cf89cb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 58cd2626, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:55:01,946-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [94fbd54] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 20:55:01,947-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [94fbd54] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 20:55:02,005-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:55:02,005-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 20:55:02,005-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 20:55:02,005-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 20:55:02,005-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 51e94b1
>2018-05-29 20:55:02,005-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 20:55:02,018-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [680e9374] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:55:02,019-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [680e9374] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2455a25
>2018-05-29 20:55:02,021-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [680e9374] FINISH, SetVdsStatusVDSCommand, log id: 2455a25
>2018-05-29 20:55:02,026-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [680e9374] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 680e9374, Job ID: ede6032b-f717-4388-81df-20aa5ccb00ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 20:55:02,030-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [680e9374] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 441071c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 20:55:02,033-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [680e9374] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 441071c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 20:55:02,043-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [2789723f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 20:55:02,044-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [2789723f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [48798a2] Autorecovering 2 hosts
>2018-05-29 21:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [48798a2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:00:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7389355f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:00:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7389355f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:00:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7389355f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:00:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7389355f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:00:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7389355f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: fb748d
>2018-05-29 21:00:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7389355f] FINISH, SetVdsStatusVDSCommand, log id: fb748d
>2018-05-29 21:00:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7389355f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:00:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7389355f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:00:00,082-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7389355f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:00:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2681ee1c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2681ee1c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2681ee1c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2681ee1c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:00:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2681ee1c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a4f641f
>2018-05-29 21:00:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2681ee1c] FINISH, SetVdsStatusVDSCommand, log id: 5a4f641f
>2018-05-29 21:00:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [2681ee1c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3f3e1346
>2018-05-29 21:00:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [2681ee1c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3f3e1346
>2018-05-29 21:00:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2681ee1c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:00:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2681ee1c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:00:02,518-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [40806728] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2d36e051
>2018-05-29 21:00:02,519-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [40806728] FINISH, GetHardwareInfoVDSCommand, log id: 2d36e051
>2018-05-29 21:00:02,533-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [22c576d7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:00:02,542-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [5382b75d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:00:02,547-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1b8d4495] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:00:02,558-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1b8d4495] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 149d937c
>2018-05-29 21:00:02,733-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6df3c88f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1f4ea2a0
>2018-05-29 21:00:02,735-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6df3c88f] FINISH, GetHardwareInfoVDSCommand, log id: 1f4ea2a0
>2018-05-29 21:00:02,747-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [676e793e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:00:02,752-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [31d9efeb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:00:02,757-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5c1e1808] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:00:02,763-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5c1e1808] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 61898d8c
>2018-05-29 21:00:02,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1b8d4495] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:00:02,940-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1b8d4495] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 149d937c
>2018-05-29 21:00:02,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1b8d4495] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:00:02,944-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1b8d4495] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:00:02,948-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1b8d4495] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:00:02,949-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1b8d4495] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:00:02,953-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1b8d4495] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:00:02,953-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1b8d4495] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:00:02,956-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1b8d4495] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:00:02,958-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1b8d4495] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1b8d4495, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:00:02,965-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1efd34e1
>2018-05-29 21:00:02,965-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:00:03,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:00:03,061-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:00:03,061-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:00:03,061-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:00:03,061-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1efd34e1
>2018-05-29 21:00:03,061-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:00:03,074-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [1cd3731b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:00:03,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1cd3731b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 37c398f
>2018-05-29 21:00:03,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1cd3731b] FINISH, SetVdsStatusVDSCommand, log id: 37c398f
>2018-05-29 21:00:03,109-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1cd3731b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1cd3731b, Job ID: e132b390-fbb5-4de4-8e7d-628bb3076175, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:00:03,114-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1cd3731b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5382b75d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:00:03,116-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1cd3731b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5382b75d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:00:03,126-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [68eeb3ae] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:00:03,128-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [68eeb3ae] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:00:03,140-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5c1e1808] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:00:03,140-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5c1e1808] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 61898d8c
>2018-05-29 21:00:03,144-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5c1e1808] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:00:03,145-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5c1e1808] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:00:03,148-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5c1e1808] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:00:03,149-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5c1e1808] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:00:03,153-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5c1e1808] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:00:03,153-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5c1e1808] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:00:03,156-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5c1e1808] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:00:03,159-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5c1e1808] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5c1e1808, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:00:03,162-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 551fc5b
>2018-05-29 21:00:03,162-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:00:03,267-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:00:03,267-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:00:03,267-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:00:03,267-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:00:03,267-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 551fc5b
>2018-05-29 21:00:03,267-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:00:03,280-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4362b678] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:00:03,281-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4362b678] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6d42692b
>2018-05-29 21:00:03,283-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4362b678] FINISH, SetVdsStatusVDSCommand, log id: 6d42692b
>2018-05-29 21:00:03,289-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4362b678] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4362b678, Job ID: ba42b361-a776-4d17-b2ec-e6393dc02109, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:00:03,293-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4362b678] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 31d9efeb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:00:03,295-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4362b678] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 31d9efeb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:00:03,305-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [24bdb77b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:00:03,307-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [24bdb77b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [7a5887be] Autorecovering 2 hosts
>2018-05-29 21:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [7a5887be] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:05:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6e5b73af] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:05:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6e5b73af] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:05:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6e5b73af] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:05:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6e5b73af] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:05:00,050-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6e5b73af] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 747f333c
>2018-05-29 21:05:00,053-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6e5b73af] FINISH, SetVdsStatusVDSCommand, log id: 747f333c
>2018-05-29 21:05:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6e5b73af] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:05:00,059-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6e5b73af] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 6e5b73af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-29 21:05:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6e5b73af] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:05:00,060-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6e5b73af] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:05:00,113-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2b5d1b18] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:05:00,119-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2b5d1b18] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:05:00,119-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2b5d1b18] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:05:00,119-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2b5d1b18] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:05:00,121-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2b5d1b18] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13627ef4
>2018-05-29 21:05:00,123-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2b5d1b18] FINISH, SetVdsStatusVDSCommand, log id: 13627ef4
>2018-05-29 21:05:00,126-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [2b5d1b18] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 68a08001
>2018-05-29 21:05:00,128-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [2b5d1b18] FINISH, SetHaMaintenanceModeVDSCommand, log id: 68a08001
>2018-05-29 21:05:00,128-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2b5d1b18] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:05:00,130-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2b5d1b18] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:05:00,729-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3faa2b03] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 466815e0
>2018-05-29 21:05:00,731-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3faa2b03] FINISH, GetHardwareInfoVDSCommand, log id: 466815e0
>2018-05-29 21:05:00,744-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [4c4a2f30] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:05:00,753-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [4199a911] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:05:00,758-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [22c2d365] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:05:00,764-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [22c2d365] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6db50aeb
>2018-05-29 21:05:00,949-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [6c0a8ead] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 787b362f
>2018-05-29 21:05:00,951-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [6c0a8ead] FINISH, GetHardwareInfoVDSCommand, log id: 787b362f
>2018-05-29 21:05:00,963-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [21275878] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:05:00,967-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [7f296c61] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:05:00,972-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [351d82a0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:05:00,978-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [351d82a0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1fe923c4
>2018-05-29 21:05:01,152-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22c2d365] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:05:01,152-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [22c2d365] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6db50aeb
>2018-05-29 21:05:01,156-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22c2d365] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:05:01,157-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [22c2d365] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:05:01,161-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22c2d365] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:05:01,161-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [22c2d365] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:05:01,165-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22c2d365] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:05:01,166-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [22c2d365] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:05:01,168-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [22c2d365] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:05:01,170-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22c2d365] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 22c2d365, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:05:01,173-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 46826498
>2018-05-29 21:05:01,173-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:05:01,270-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:05:01,270-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:05:01,270-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:05:01,270-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:05:01,270-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 46826498
>2018-05-29 21:05:01,270-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:05:01,283-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [306f7f3e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:05:01,284-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [306f7f3e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 62331714
>2018-05-29 21:05:01,286-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [306f7f3e] FINISH, SetVdsStatusVDSCommand, log id: 62331714
>2018-05-29 21:05:01,315-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [306f7f3e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 306f7f3e, Job ID: b4def737-96ef-4044-8a4d-a3a0c7a3cc74, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:05:01,320-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [306f7f3e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4199a911, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:05:01,322-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [306f7f3e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4199a911, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:05:01,333-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [679a8468] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:05:01,334-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [679a8468] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:05:02,349-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [351d82a0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:05:02,349-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [351d82a0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1fe923c4
>2018-05-29 21:05:02,353-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [351d82a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:05:02,353-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [351d82a0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:05:02,357-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [351d82a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:05:02,358-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [351d82a0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:05:02,362-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [351d82a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:05:02,363-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [351d82a0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:05:02,366-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [351d82a0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:05:02,369-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [351d82a0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 351d82a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:05:02,372-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 57869e5f
>2018-05-29 21:05:02,372-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:05:02,471-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:05:02,471-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:05:02,471-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:05:02,471-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:05:02,471-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 57869e5f
>2018-05-29 21:05:02,471-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:05:02,484-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [2071067a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:05:02,485-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2071067a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b1f9c5a
>2018-05-29 21:05:02,487-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2071067a] FINISH, SetVdsStatusVDSCommand, log id: 1b1f9c5a
>2018-05-29 21:05:02,493-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2071067a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2071067a, Job ID: 0ff06924-9827-4384-aa66-587e2a12934b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:05:02,498-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2071067a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7f296c61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:05:02,500-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2071067a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7f296c61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:05:02,511-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [ebc71bc] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:05:02,512-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [ebc71bc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:10:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [1bac5d4] Autorecovering 2 hosts
>2018-05-29 21:10:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [1bac5d4] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:10:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [631b1ef3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:10:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [631b1ef3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:10:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [631b1ef3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:10:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [631b1ef3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:10:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [631b1ef3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7326792e
>2018-05-29 21:10:00,045-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [631b1ef3] FINISH, SetVdsStatusVDSCommand, log id: 7326792e
>2018-05-29 21:10:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [631b1ef3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:10:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [631b1ef3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:10:00,048-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [631b1ef3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:10:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4bdddea3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:10:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4bdddea3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:10:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4bdddea3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:10:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4bdddea3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:10:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4bdddea3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1edf3bc9
>2018-05-29 21:10:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4bdddea3] FINISH, SetVdsStatusVDSCommand, log id: 1edf3bc9
>2018-05-29 21:10:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [4bdddea3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 482c4323
>2018-05-29 21:10:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [4bdddea3] FINISH, SetHaMaintenanceModeVDSCommand, log id: 482c4323
>2018-05-29 21:10:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4bdddea3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:10:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4bdddea3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:10:00,213-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [65b59aff] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2c6975c6
>2018-05-29 21:10:00,215-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [65b59aff] FINISH, GetHardwareInfoVDSCommand, log id: 2c6975c6
>2018-05-29 21:10:00,227-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [5eae1697] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:10:00,235-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [79df117c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:10:00,241-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [539fad5c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:10:00,252-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [539fad5c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6999aa1a
>2018-05-29 21:10:00,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [539fad5c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:10:00,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [539fad5c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6999aa1a
>2018-05-29 21:10:00,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [539fad5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:10:00,632-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [539fad5c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:10:00,635-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [539fad5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:10:00,636-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [539fad5c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:10:00,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [539fad5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:10:00,640-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [539fad5c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:10:00,643-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [539fad5c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:10:00,645-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [539fad5c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 539fad5c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:10:00,652-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: ca0c4f1
>2018-05-29 21:10:00,652-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:10:00,753-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:10:00,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:10:00,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:10:00,753-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:10:00,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: ca0c4f1
>2018-05-29 21:10:00,753-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:10:00,766-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [74743ba5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:10:00,767-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [74743ba5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1c2b36e5
>2018-05-29 21:10:00,769-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [74743ba5] FINISH, SetVdsStatusVDSCommand, log id: 1c2b36e5
>2018-05-29 21:10:00,777-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [74743ba5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 74743ba5, Job ID: 319e10ff-cd19-411a-b67f-98a6355056cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:10:00,781-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [74743ba5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 79df117c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:10:00,784-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [74743ba5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 79df117c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:10:00,794-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [4414b58b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:10:00,795-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [4414b58b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:10:01,945-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5231a638] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4a998fc7
>2018-05-29 21:10:01,947-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5231a638] FINISH, GetHardwareInfoVDSCommand, log id: 4a998fc7
>2018-05-29 21:10:01,960-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [1b8e08ef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:10:01,964-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [1d78fb20] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:10:01,970-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [262b095] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:10:01,981-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [262b095] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22e11bd2
>2018-05-29 21:10:02,356-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [262b095] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:10:02,356-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [262b095] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 22e11bd2
>2018-05-29 21:10:02,360-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [262b095] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:10:02,361-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [262b095] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:10:02,365-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [262b095] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:10:02,365-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [262b095] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:10:02,369-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [262b095] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:10:02,370-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [262b095] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:10:02,373-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [262b095] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:10:02,375-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [262b095] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 262b095, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:10:02,378-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7ab004fd
>2018-05-29 21:10:02,378-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:10:02,472-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:10:02,472-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:10:02,472-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:10:02,472-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:10:02,472-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7ab004fd
>2018-05-29 21:10:02,472-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:10:02,485-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [c66cf12] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:10:02,486-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [c66cf12] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6899ca56
>2018-05-29 21:10:02,489-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [c66cf12] FINISH, SetVdsStatusVDSCommand, log id: 6899ca56
>2018-05-29 21:10:02,494-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [c66cf12] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: c66cf12, Job ID: 3ff620e4-3a9d-489f-9806-c2ba82d73264, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:10:02,499-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [c66cf12] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d78fb20, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:10:02,501-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [c66cf12] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d78fb20, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:10:02,511-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [224aaddd] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:10:02,512-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [224aaddd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [381ae06f] Autorecovering 2 hosts
>2018-05-29 21:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [381ae06f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:15:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [780ce812] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:15:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [780ce812] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:15:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [780ce812] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:15:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [780ce812] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:15:00,050-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [780ce812] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 51eb280a
>2018-05-29 21:15:00,053-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [780ce812] FINISH, SetVdsStatusVDSCommand, log id: 51eb280a
>2018-05-29 21:15:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [780ce812] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:15:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [780ce812] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:15:00,056-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [780ce812] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:15:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6aa0af87] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:15:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6aa0af87] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:15:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6aa0af87] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:15:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6aa0af87] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:15:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6aa0af87] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4767cf6
>2018-05-29 21:15:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6aa0af87] FINISH, SetVdsStatusVDSCommand, log id: 4767cf6
>2018-05-29 21:15:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [6aa0af87] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 300867f9
>2018-05-29 21:15:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [6aa0af87] FINISH, SetHaMaintenanceModeVDSCommand, log id: 300867f9
>2018-05-29 21:15:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6aa0af87] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:15:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6aa0af87] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:15:01,504-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [37d2b514] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5be3cefc
>2018-05-29 21:15:01,505-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [37d2b514] FINISH, GetHardwareInfoVDSCommand, log id: 5be3cefc
>2018-05-29 21:15:01,518-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [10362b06] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:15:01,526-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [4b869451] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:15:01,531-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [489bd871] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:15:01,537-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [489bd871] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3bc07f7e
>2018-05-29 21:15:01,905-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [489bd871] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:15:01,905-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [489bd871] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3bc07f7e
>2018-05-29 21:15:01,909-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [489bd871] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:15:01,910-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [489bd871] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:15:01,913-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [489bd871] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:15:01,914-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [489bd871] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:15:01,918-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [489bd871] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:15:01,918-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [489bd871] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:15:01,921-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [489bd871] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:15:01,923-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [489bd871] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 489bd871, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:15:01,930-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 165d334a
>2018-05-29 21:15:01,930-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:15:02,026-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:15:02,026-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:15:02,026-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:15:02,026-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:15:02,026-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 165d334a
>2018-05-29 21:15:02,026-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:15:02,039-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [438f02a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:15:02,040-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [438f02a6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 475528f8
>2018-05-29 21:15:02,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [438f02a6] FINISH, SetVdsStatusVDSCommand, log id: 475528f8
>2018-05-29 21:15:02,075-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [438f02a6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 438f02a6, Job ID: 9e42e485-c1c8-4ceb-a0c3-c95099ebfdc6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:15:02,079-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [438f02a6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4b869451, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:15:02,081-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [438f02a6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4b869451, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:15:02,092-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [59b6d365] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:15:02,093-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [59b6d365] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:15:03,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1d7b9103] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 66014035
>2018-05-29 21:15:03,180-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1d7b9103] FINISH, GetHardwareInfoVDSCommand, log id: 66014035
>2018-05-29 21:15:03,192-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [343dfa6c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:15:03,197-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [77035c56] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:15:03,202-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [12737317] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:15:03,208-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [12737317] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 65b596b2
>2018-05-29 21:15:03,584-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12737317] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:15:03,584-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [12737317] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 65b596b2
>2018-05-29 21:15:03,587-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12737317] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:15:03,588-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [12737317] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:15:03,592-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12737317] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:15:03,593-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [12737317] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:15:03,596-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12737317] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:15:03,597-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [12737317] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:15:03,600-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [12737317] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:15:03,602-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12737317] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 12737317, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:15:03,605-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 215e0aa
>2018-05-29 21:15:03,605-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:15:03,702-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:15:03,702-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:15:03,702-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:15:03,702-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:15:03,702-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 215e0aa
>2018-05-29 21:15:03,702-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:15:03,715-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [4f9b8501] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:15:03,716-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4f9b8501] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50a7bb7a
>2018-05-29 21:15:03,718-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4f9b8501] FINISH, SetVdsStatusVDSCommand, log id: 50a7bb7a
>2018-05-29 21:15:03,724-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f9b8501] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4f9b8501, Job ID: d8e2b6e5-ecd3-4714-862d-3fe148a6ba10, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:15:03,729-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f9b8501] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 77035c56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:15:03,731-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f9b8501] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 77035c56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:15:03,742-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [6ce398db] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:15:03,743-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [6ce398db] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:20:00,007-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [543eb6ea] Autorecovering 2 hosts
>2018-05-29 21:20:00,007-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [543eb6ea] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:20:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6e2a39c1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6e2a39c1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6e2a39c1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6e2a39c1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:20:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6e2a39c1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e90b5f6
>2018-05-29 21:20:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6e2a39c1] FINISH, SetVdsStatusVDSCommand, log id: 5e90b5f6
>2018-05-29 21:20:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6e2a39c1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:20:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6e2a39c1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:20:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6e2a39c1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:20:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b5bd4b1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:20:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b5bd4b1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:20:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b5bd4b1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:20:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b5bd4b1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:20:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7b5bd4b1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 454a1b8c
>2018-05-29 21:20:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7b5bd4b1] FINISH, SetVdsStatusVDSCommand, log id: 454a1b8c
>2018-05-29 21:20:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7b5bd4b1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 647ec951
>2018-05-29 21:20:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7b5bd4b1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 647ec951
>2018-05-29 21:20:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b5bd4b1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:20:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b5bd4b1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:20:01,349-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1dde2490] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7bb697b2
>2018-05-29 21:20:01,351-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1dde2490] FINISH, GetHardwareInfoVDSCommand, log id: 7bb697b2
>2018-05-29 21:20:01,364-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [4aa6d213] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:20:01,372-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [271dd3e0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:20:01,377-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [364463d2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:20:01,383-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [364463d2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5ef27ca8
>2018-05-29 21:20:01,763-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [364463d2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:20:01,763-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [364463d2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5ef27ca8
>2018-05-29 21:20:01,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [364463d2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:20:01,768-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [364463d2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:20:01,772-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [364463d2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:20:01,772-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [364463d2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:20:01,776-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [364463d2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:20:01,777-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [364463d2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:20:01,779-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [364463d2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:20:01,782-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [364463d2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 364463d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:20:01,784-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4ce5a040
>2018-05-29 21:20:01,784-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:20:01,884-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:20:01,885-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:20:01,885-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:20:01,885-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:20:01,885-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4ce5a040
>2018-05-29 21:20:01,885-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:20:01,898-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [3d1acb4f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:20:01,899-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3d1acb4f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2cc20b8d
>2018-05-29 21:20:01,901-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3d1acb4f] FINISH, SetVdsStatusVDSCommand, log id: 2cc20b8d
>2018-05-29 21:20:01,932-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3d1acb4f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3d1acb4f, Job ID: 2e9cb2d4-c695-43dd-bf64-e498da7d8c97, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:20:01,936-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3d1acb4f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 271dd3e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:20:01,938-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3d1acb4f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 271dd3e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:20:01,948-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [5419ad82] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:20:01,949-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [5419ad82] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:20:02,943-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2bc38e4b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 324a8bfe
>2018-05-29 21:20:02,945-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2bc38e4b] FINISH, GetHardwareInfoVDSCommand, log id: 324a8bfe
>2018-05-29 21:20:02,957-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [6cc600e1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:20:02,962-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [55772243] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:20:02,967-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [39803d00] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:20:02,974-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [39803d00] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3c92f500
>2018-05-29 21:20:03,351-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39803d00] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:20:03,351-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [39803d00] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3c92f500
>2018-05-29 21:20:03,355-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39803d00] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:20:03,355-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [39803d00] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:20:03,359-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39803d00] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:20:03,360-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [39803d00] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:20:03,363-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39803d00] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:20:03,364-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [39803d00] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:20:03,367-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [39803d00] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:20:03,369-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39803d00] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 39803d00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:20:03,372-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3970c7bc
>2018-05-29 21:20:03,372-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:20:03,467-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:20:03,467-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:20:03,467-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:20:03,467-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:20:03,467-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3970c7bc
>2018-05-29 21:20:03,467-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:20:03,480-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [178e6aea] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:20:03,481-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [178e6aea] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 411bf957
>2018-05-29 21:20:03,483-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [178e6aea] FINISH, SetVdsStatusVDSCommand, log id: 411bf957
>2018-05-29 21:20:03,488-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [178e6aea] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 178e6aea, Job ID: 93c31075-3ebd-46c4-b757-e1fb45963eb9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:20:03,493-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [178e6aea] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 55772243, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:20:03,495-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [178e6aea] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 55772243, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:20:03,505-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6db011a8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:20:03,506-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6db011a8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [50ea0434] Autorecovering 2 hosts
>2018-05-29 21:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [50ea0434] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:25:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79c09aa0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:25:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79c09aa0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:25:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79c09aa0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:25:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79c09aa0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:25:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [79c09aa0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 38864050
>2018-05-29 21:25:00,080-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [79c09aa0] FINISH, SetVdsStatusVDSCommand, log id: 38864050
>2018-05-29 21:25:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79c09aa0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:25:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79c09aa0] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:25:00,083-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [79c09aa0] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [60cff827] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:25:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [60cff827] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:25:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [60cff827] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:25:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [60cff827] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:25:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [60cff827] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 409e0580
>2018-05-29 21:25:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [60cff827] FINISH, SetVdsStatusVDSCommand, log id: 409e0580
>2018-05-29 21:25:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [60cff827] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2f99513b
>2018-05-29 21:25:00,104-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [60cff827] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2f99513b
>2018-05-29 21:25:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [60cff827] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:25:00,106-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [60cff827] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:25:01,137-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7da91647] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 212ccc81
>2018-05-29 21:25:01,139-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7da91647] FINISH, GetHardwareInfoVDSCommand, log id: 212ccc81
>2018-05-29 21:25:01,153-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [6699b168] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:25:01,161-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [18a02a44] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:25:01,166-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5a8e2d48] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:25:01,177-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5a8e2d48] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2cf90112
>2018-05-29 21:25:02,551-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5a8e2d48] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:25:02,552-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5a8e2d48] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2cf90112
>2018-05-29 21:25:02,556-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5a8e2d48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:25:02,556-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5a8e2d48] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:25:02,561-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5a8e2d48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:25:02,561-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5a8e2d48] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:25:02,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5a8e2d48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:25:02,566-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5a8e2d48] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:25:02,570-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5a8e2d48] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:25:02,574-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5a8e2d48] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5a8e2d48, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:25:02,580-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 38088366
>2018-05-29 21:25:02,580-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:25:02,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:25:02,680-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:25:02,680-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:25:02,680-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:25:02,680-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 38088366
>2018-05-29 21:25:02,680-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:25:02,708-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [4e953eb1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:25:02,710-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4e953eb1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6a1ec433
>2018-05-29 21:25:02,712-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4e953eb1] FINISH, SetVdsStatusVDSCommand, log id: 6a1ec433
>2018-05-29 21:25:02,746-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4e953eb1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4e953eb1, Job ID: 972b4e5e-4d57-48d7-b4b8-de17ce379f3b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:25:02,751-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4e953eb1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 18a02a44, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:25:02,754-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4e953eb1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 18a02a44, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:25:02,765-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [532c52b9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:25:02,766-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [532c52b9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:25:02,835-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4cf31885] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1f80074f
>2018-05-29 21:25:02,837-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4cf31885] FINISH, GetHardwareInfoVDSCommand, log id: 1f80074f
>2018-05-29 21:25:02,857-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [442ec85c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:25:02,865-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [68fccb6e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:25:02,870-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4ebb7b03] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:25:02,882-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4ebb7b03] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6baced66
>2018-05-29 21:25:03,264-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ebb7b03] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:25:03,265-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4ebb7b03] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6baced66
>2018-05-29 21:25:03,268-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ebb7b03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:25:03,269-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4ebb7b03] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:25:03,273-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ebb7b03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:25:03,273-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4ebb7b03] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:25:03,278-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ebb7b03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:25:03,278-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4ebb7b03] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:25:03,281-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4ebb7b03] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:25:03,284-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ebb7b03] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4ebb7b03, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:25:03,286-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: ba176c9
>2018-05-29 21:25:03,286-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:25:03,382-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:25:03,382-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:25:03,382-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:25:03,382-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:25:03,382-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: ba176c9
>2018-05-29 21:25:03,382-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:25:03,396-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [325ab866] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:25:03,397-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [325ab866] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d91c7ff
>2018-05-29 21:25:03,399-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [325ab866] FINISH, SetVdsStatusVDSCommand, log id: 2d91c7ff
>2018-05-29 21:25:03,405-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [325ab866] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 325ab866, Job ID: 0c390c9d-2d6e-44c8-8706-89e757c61204, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:25:03,410-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [325ab866] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 68fccb6e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:25:03,412-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [325ab866] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 68fccb6e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:25:03,423-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [73cee93d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:25:03,424-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [73cee93d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [e7174dd] Autorecovering 2 hosts
>2018-05-29 21:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [e7174dd] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:30:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2663b02f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2663b02f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2663b02f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2663b02f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:30:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2663b02f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f073be0
>2018-05-29 21:30:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2663b02f] FINISH, SetVdsStatusVDSCommand, log id: 1f073be0
>2018-05-29 21:30:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2663b02f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:30:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2663b02f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:30:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [2663b02f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:30:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [24f0f6a3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:30:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [24f0f6a3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:30:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [24f0f6a3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:30:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [24f0f6a3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:30:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [24f0f6a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bc6b66a
>2018-05-29 21:30:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [24f0f6a3] FINISH, SetVdsStatusVDSCommand, log id: 3bc6b66a
>2018-05-29 21:30:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [24f0f6a3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5457e97f
>2018-05-29 21:30:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [24f0f6a3] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5457e97f
>2018-05-29 21:30:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [24f0f6a3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:30:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [24f0f6a3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:30:00,480-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [541898f0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 22b27601
>2018-05-29 21:30:00,482-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [541898f0] FINISH, GetHardwareInfoVDSCommand, log id: 22b27601
>2018-05-29 21:30:00,495-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [631d05ae] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:30:00,503-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [70451c85] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:30:00,509-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [530a9a60] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:30:00,516-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [530a9a60] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 61c71ff9
>2018-05-29 21:30:00,898-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [530a9a60] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:30:00,898-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [530a9a60] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 61c71ff9
>2018-05-29 21:30:00,902-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [530a9a60] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:30:00,903-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [530a9a60] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:30:00,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [530a9a60] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:30:00,908-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [530a9a60] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:30:00,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [530a9a60] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:30:00,912-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [530a9a60] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:30:00,916-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [530a9a60] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:30:00,918-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [530a9a60] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 530a9a60, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:30:00,926-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5012276a
>2018-05-29 21:30:00,926-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:30:01,024-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:30:01,024-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:30:01,024-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:30:01,024-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:30:01,024-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5012276a
>2018-05-29 21:30:01,024-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:30:01,064-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [23e358af] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:30:01,065-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [761e4b5e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 19703a2e
>2018-05-29 21:30:01,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [23e358af] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52b81dfc
>2018-05-29 21:30:01,067-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [761e4b5e] FINISH, GetHardwareInfoVDSCommand, log id: 19703a2e
>2018-05-29 21:30:01,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [23e358af] FINISH, SetVdsStatusVDSCommand, log id: 52b81dfc
>2018-05-29 21:30:01,081-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [37ea5e9b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:30:01,090-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [308fb2f5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:30:01,096-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [490679b5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:30:01,103-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [23e358af] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 23e358af, Job ID: 54d4746e-9621-42bf-b9bc-c3c5f4eba343, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:30:01,108-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [23e358af] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 70451c85, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:30:01,109-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [490679b5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: d3d7aa6
>2018-05-29 21:30:01,111-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [23e358af] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 70451c85, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:30:01,122-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [19473fcf] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:30:01,123-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [19473fcf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:30:01,488-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490679b5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:30:01,488-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [490679b5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: d3d7aa6
>2018-05-29 21:30:01,491-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490679b5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:30:01,492-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [490679b5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:30:01,496-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490679b5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:30:01,496-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [490679b5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:30:01,500-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490679b5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:30:01,501-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [490679b5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:30:01,505-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [490679b5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:30:01,507-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490679b5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 490679b5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:30:01,510-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 12814588
>2018-05-29 21:30:01,510-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:30:01,605-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:30:01,605-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:30:01,605-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:30:01,605-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:30:01,605-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 12814588
>2018-05-29 21:30:01,605-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:30:01,618-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [49de8836] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:30:01,619-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [49de8836] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4e18e0fb
>2018-05-29 21:30:01,622-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [49de8836] FINISH, SetVdsStatusVDSCommand, log id: 4e18e0fb
>2018-05-29 21:30:01,630-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [49de8836] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 49de8836, Job ID: 3c192c7a-c4d9-4b5b-92e7-ab6cefdcf115, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:30:01,635-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [49de8836] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 308fb2f5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:30:01,637-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [49de8836] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 308fb2f5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:30:01,647-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [41bdf60] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:30:01,648-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [41bdf60] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [536afe6d] Autorecovering 2 hosts
>2018-05-29 21:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [536afe6d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:35:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5886dae9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5886dae9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5886dae9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5886dae9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:35:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5886dae9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ad8035d
>2018-05-29 21:35:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5886dae9] FINISH, SetVdsStatusVDSCommand, log id: 3ad8035d
>2018-05-29 21:35:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5886dae9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:35:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5886dae9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:35:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5886dae9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:35:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [14f491a0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:35:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [14f491a0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:35:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [14f491a0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:35:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [14f491a0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:35:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [14f491a0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 29e40cf1
>2018-05-29 21:35:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [14f491a0] FINISH, SetVdsStatusVDSCommand, log id: 29e40cf1
>2018-05-29 21:35:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [14f491a0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 724b8282
>2018-05-29 21:35:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [14f491a0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 724b8282
>2018-05-29 21:35:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [14f491a0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:35:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [14f491a0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:35:01,873-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3e02b3cb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 580fc5dd
>2018-05-29 21:35:01,874-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3e02b3cb] FINISH, GetHardwareInfoVDSCommand, log id: 580fc5dd
>2018-05-29 21:35:01,887-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [14dc8e2c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:35:01,896-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [5335aa18] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:35:01,901-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4749cca] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:35:01,912-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4749cca] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2d4ee2ce
>2018-05-29 21:35:02,259-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3f6f956b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 26c81057
>2018-05-29 21:35:02,261-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3f6f956b] FINISH, GetHardwareInfoVDSCommand, log id: 26c81057
>2018-05-29 21:35:02,273-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [27f2c8dc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:35:02,277-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5f88fefa] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:35:02,283-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [771a90e2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:35:02,289-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [771a90e2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5ed7a800
>2018-05-29 21:35:02,674-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [771a90e2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:35:02,674-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [771a90e2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5ed7a800
>2018-05-29 21:35:02,678-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [771a90e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:35:02,679-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [771a90e2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:35:02,683-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [771a90e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:35:02,683-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [771a90e2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:35:02,687-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [771a90e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:35:02,688-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [771a90e2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:35:02,691-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [771a90e2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:35:02,693-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [771a90e2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 771a90e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:35:02,700-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1cd8d47c
>2018-05-29 21:35:02,700-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:35:02,798-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:35:02,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:35:02,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:35:02,798-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:35:02,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1cd8d47c
>2018-05-29 21:35:02,798-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:35:02,811-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [471f819c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:35:02,813-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [471f819c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7ebbaa72
>2018-05-29 21:35:02,815-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [471f819c] FINISH, SetVdsStatusVDSCommand, log id: 7ebbaa72
>2018-05-29 21:35:02,848-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [471f819c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 471f819c, Job ID: fa0c6696-bdbe-4b43-addc-8a2ff143a4f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:35:02,853-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [471f819c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5f88fefa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:35:02,855-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [471f819c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5f88fefa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:35:02,866-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [7728beec] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:35:02,867-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7728beec] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:35:03,281-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4749cca] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:35:03,281-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4749cca] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2d4ee2ce
>2018-05-29 21:35:03,285-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4749cca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:35:03,286-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4749cca] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:35:03,290-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4749cca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:35:03,291-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4749cca] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:35:03,294-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4749cca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:35:03,295-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4749cca] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:35:03,299-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4749cca] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:35:03,302-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4749cca] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4749cca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:35:03,305-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 62216795
>2018-05-29 21:35:03,305-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:35:03,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:35:03,403-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:35:03,403-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:35:03,403-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:35:03,403-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 62216795
>2018-05-29 21:35:03,403-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:35:03,416-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [33767cf5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:35:03,418-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [33767cf5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42e99742
>2018-05-29 21:35:03,420-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [33767cf5] FINISH, SetVdsStatusVDSCommand, log id: 42e99742
>2018-05-29 21:35:03,425-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [33767cf5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 33767cf5, Job ID: 0c265509-aaa6-4306-a528-805b9e953fca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:35:03,430-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [33767cf5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5335aa18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:35:03,432-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [33767cf5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5335aa18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:35:03,442-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [46b6edc4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:35:03,443-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [46b6edc4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [11dfbe93] Autorecovering 2 hosts
>2018-05-29 21:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [11dfbe93] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:40:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [53abf973] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [53abf973] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [53abf973] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [53abf973] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:40:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [53abf973] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2080f57a
>2018-05-29 21:40:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [53abf973] FINISH, SetVdsStatusVDSCommand, log id: 2080f57a
>2018-05-29 21:40:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [53abf973] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:40:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [53abf973] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:40:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [53abf973] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:40:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70bda03a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:40:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70bda03a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:40:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70bda03a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:40:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70bda03a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:40:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [70bda03a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 413dbf6f
>2018-05-29 21:40:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [70bda03a] FINISH, SetVdsStatusVDSCommand, log id: 413dbf6f
>2018-05-29 21:40:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [70bda03a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4a155340
>2018-05-29 21:40:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [70bda03a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4a155340
>2018-05-29 21:40:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70bda03a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:40:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70bda03a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:40:00,536-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [50278757] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7eae43d0
>2018-05-29 21:40:00,537-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [50278757] FINISH, GetHardwareInfoVDSCommand, log id: 7eae43d0
>2018-05-29 21:40:00,551-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [4779f2ee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:40:00,559-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [19405a6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:40:00,565-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3e5c8c5b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:40:00,578-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3e5c8c5b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5a2edb5a
>2018-05-29 21:40:00,964-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e5c8c5b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:40:00,964-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3e5c8c5b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5a2edb5a
>2018-05-29 21:40:00,968-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e5c8c5b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:40:00,969-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3e5c8c5b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:40:00,973-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e5c8c5b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:40:00,974-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3e5c8c5b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:40:00,977-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e5c8c5b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:40:00,978-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3e5c8c5b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:40:00,982-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3e5c8c5b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:40:00,985-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e5c8c5b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3e5c8c5b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:40:00,992-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 11ebe761
>2018-05-29 21:40:00,992-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:40:01,089-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7489c281] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 64768251
>2018-05-29 21:40:01,090-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:40:01,090-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:40:01,090-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:40:01,090-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:40:01,090-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 11ebe761
>2018-05-29 21:40:01,090-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:40:01,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7489c281] FINISH, GetHardwareInfoVDSCommand, log id: 64768251
>2018-05-29 21:40:01,105-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [163e0325] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:40:01,106-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [6107ce39] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:40:01,107-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6107ce39] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d233f7d
>2018-05-29 21:40:01,109-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6107ce39] FINISH, SetVdsStatusVDSCommand, log id: 2d233f7d
>2018-05-29 21:40:01,114-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [7f497a91] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:40:01,119-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [54d21994] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:40:01,132-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [54d21994] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 14bcf709
>2018-05-29 21:40:01,143-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6107ce39] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6107ce39, Job ID: b8b4ee37-3eb9-4dc4-a225-abe8a532424f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:40:01,147-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6107ce39] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 19405a6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:40:01,149-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6107ce39] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 19405a6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:40:01,160-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [13817581] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:40:01,161-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [13817581] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:40:01,503-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54d21994] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:40:01,503-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [54d21994] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 14bcf709
>2018-05-29 21:40:01,507-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54d21994] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:40:01,508-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [54d21994] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:40:01,512-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54d21994] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:40:01,512-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [54d21994] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:40:01,516-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54d21994] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:40:01,517-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [54d21994] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:40:01,520-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [54d21994] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:40:01,522-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54d21994] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 54d21994, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:40:01,528-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 513d7d63
>2018-05-29 21:40:01,528-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:40:01,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:40:01,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:40:01,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:40:01,627-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:40:01,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 513d7d63
>2018-05-29 21:40:01,627-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:40:01,640-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [6e2cd4ac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:40:01,641-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6e2cd4ac] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32fa64da
>2018-05-29 21:40:01,643-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6e2cd4ac] FINISH, SetVdsStatusVDSCommand, log id: 32fa64da
>2018-05-29 21:40:01,648-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6e2cd4ac] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6e2cd4ac, Job ID: 4d988bad-a574-40a3-a3ed-14c47b5d8ab4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:40:01,653-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6e2cd4ac] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7f497a91, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:40:01,655-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6e2cd4ac] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7f497a91, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:40:01,665-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2c088b85] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:40:01,666-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2c088b85] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7389da8a] Autorecovering 2 hosts
>2018-05-29 21:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7389da8a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:45:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [31ae02cd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [31ae02cd] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [31ae02cd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [31ae02cd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:45:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [31ae02cd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e425707
>2018-05-29 21:45:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [31ae02cd] FINISH, SetVdsStatusVDSCommand, log id: 5e425707
>2018-05-29 21:45:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [31ae02cd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:45:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [31ae02cd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:45:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [31ae02cd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:45:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5c99c35c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5c99c35c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5c99c35c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5c99c35c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:45:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5c99c35c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 626bd472
>2018-05-29 21:45:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5c99c35c] FINISH, SetVdsStatusVDSCommand, log id: 626bd472
>2018-05-29 21:45:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [5c99c35c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 53dcae7f
>2018-05-29 21:45:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [5c99c35c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 53dcae7f
>2018-05-29 21:45:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5c99c35c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:45:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5c99c35c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:45:02,059-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [8f559e7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6620f23a
>2018-05-29 21:45:02,061-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [8f559e7] FINISH, GetHardwareInfoVDSCommand, log id: 6620f23a
>2018-05-29 21:45:02,074-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [24175f8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:45:02,079-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [23c4a17b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:45:02,084-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [76a31956] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:45:02,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [76a31956] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2455f89
>2018-05-29 21:45:02,392-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3fd7dbd0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 21cfb32c
>2018-05-29 21:45:02,393-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3fd7dbd0] FINISH, GetHardwareInfoVDSCommand, log id: 21cfb32c
>2018-05-29 21:45:02,406-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [2422a435] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:45:02,410-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [1c2b1033] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:45:02,415-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6036e22b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:45:02,421-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6036e22b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 774b0816
>2018-05-29 21:45:02,466-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [76a31956] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:45:02,466-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [76a31956] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2455f89
>2018-05-29 21:45:02,470-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [76a31956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:45:02,471-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [76a31956] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:45:02,474-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [76a31956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:45:02,475-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [76a31956] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:45:02,479-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [76a31956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:45:02,479-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [76a31956] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:45:02,482-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [76a31956] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:45:02,485-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [76a31956] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 76a31956, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:45:02,487-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2e0af3bd
>2018-05-29 21:45:02,488-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:45:02,582-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:45:02,582-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:45:02,582-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:45:02,582-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:45:02,582-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2e0af3bd
>2018-05-29 21:45:02,582-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:45:02,595-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [6e28fe8e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:45:02,596-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6e28fe8e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 34021b6a
>2018-05-29 21:45:02,598-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6e28fe8e] FINISH, SetVdsStatusVDSCommand, log id: 34021b6a
>2018-05-29 21:45:02,627-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6e28fe8e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6e28fe8e, Job ID: 40fe12ca-f7f9-46b1-a4de-757095cf38b5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:45:02,632-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6e28fe8e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 23c4a17b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:45:02,633-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6e28fe8e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 23c4a17b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:45:02,644-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [6580b09f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:45:02,645-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [6580b09f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:45:02,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6036e22b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:45:02,792-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6036e22b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 774b0816
>2018-05-29 21:45:02,796-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6036e22b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:45:02,796-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6036e22b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:45:02,800-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6036e22b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:45:02,801-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6036e22b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:45:02,804-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6036e22b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:45:02,805-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6036e22b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:45:02,809-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6036e22b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:45:02,811-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6036e22b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6036e22b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:45:02,814-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2e946201
>2018-05-29 21:45:02,814-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:45:02,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:45:02,912-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:45:02,912-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:45:02,912-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:45:02,912-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2e946201
>2018-05-29 21:45:02,912-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:45:02,925-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [534fccb7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:45:02,926-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [534fccb7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49d04571
>2018-05-29 21:45:02,928-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [534fccb7] FINISH, SetVdsStatusVDSCommand, log id: 49d04571
>2018-05-29 21:45:02,933-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [534fccb7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 534fccb7, Job ID: 625efb00-78b0-4278-9e03-e9bbcdeb118c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:45:02,938-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [534fccb7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1c2b1033, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:45:02,940-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [534fccb7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1c2b1033, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:45:02,950-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [5b3d0daf] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:45:02,952-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [5b3d0daf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [4ab21ffa] Autorecovering 2 hosts
>2018-05-29 21:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [4ab21ffa] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:50:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5e9f04a7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:50:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5e9f04a7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:50:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5e9f04a7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:50:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5e9f04a7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:50:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5e9f04a7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 570a9e9e
>2018-05-29 21:50:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5e9f04a7] FINISH, SetVdsStatusVDSCommand, log id: 570a9e9e
>2018-05-29 21:50:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5e9f04a7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:50:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5e9f04a7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:50:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5e9f04a7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:50:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4391eeb9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:50:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4391eeb9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:50:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4391eeb9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:50:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4391eeb9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:50:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4391eeb9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3333cc95
>2018-05-29 21:50:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4391eeb9] FINISH, SetVdsStatusVDSCommand, log id: 3333cc95
>2018-05-29 21:50:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [4391eeb9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 414c7fcc
>2018-05-29 21:50:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [4391eeb9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 414c7fcc
>2018-05-29 21:50:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4391eeb9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:50:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4391eeb9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:50:00,226-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [77b048da] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 18d8854c
>2018-05-29 21:50:00,228-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [77b048da] FINISH, GetHardwareInfoVDSCommand, log id: 18d8854c
>2018-05-29 21:50:00,241-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [1c98227c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:50:00,250-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [62ca2f6d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:50:00,256-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5041cd15] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:50:00,267-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5041cd15] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 26827452
>2018-05-29 21:50:00,625-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [404db175] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 55ed9423
>2018-05-29 21:50:00,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [404db175] FINISH, GetHardwareInfoVDSCommand, log id: 55ed9423
>2018-05-29 21:50:00,640-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [1a28b118] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:50:00,647-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5041cd15] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:50:00,647-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5041cd15] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 26827452
>2018-05-29 21:50:00,649-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [15cd92b0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:50:00,653-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5041cd15] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:50:00,653-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5041cd15] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:50:00,655-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [73e3908e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:50:00,657-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5041cd15] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:50:00,658-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5041cd15] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:50:00,662-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5041cd15] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:50:00,663-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5041cd15] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:50:00,666-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5041cd15] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:50:00,668-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [73e3908e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 62b7d0b8
>2018-05-29 21:50:00,670-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5041cd15] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5041cd15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:50:00,676-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 23513564
>2018-05-29 21:50:00,676-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:50:00,775-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:50:00,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:50:00,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:50:00,775-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:50:00,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 23513564
>2018-05-29 21:50:00,775-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:50:00,791-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [19313a95] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:50:00,792-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [19313a95] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7a3b0d2d
>2018-05-29 21:50:00,794-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [19313a95] FINISH, SetVdsStatusVDSCommand, log id: 7a3b0d2d
>2018-05-29 21:50:00,825-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19313a95] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 19313a95, Job ID: abd9997b-0647-4480-9fcd-546d8e8ac667, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:50:00,830-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19313a95] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 62ca2f6d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:50:00,832-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19313a95] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 62ca2f6d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:50:00,843-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [50f97f50] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:50:00,844-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [50f97f50] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:50:01,048-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73e3908e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:50:01,048-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [73e3908e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 62b7d0b8
>2018-05-29 21:50:01,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73e3908e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:50:01,052-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [73e3908e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:50:01,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73e3908e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:50:01,057-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [73e3908e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:50:01,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73e3908e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:50:01,062-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [73e3908e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:50:01,065-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [73e3908e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:50:01,067-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73e3908e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 73e3908e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:50:01,071-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 51522633
>2018-05-29 21:50:01,071-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:50:01,171-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:50:01,171-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:50:01,171-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:50:01,172-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:50:01,172-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 51522633
>2018-05-29 21:50:01,172-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:50:01,184-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [7e89c08f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:50:01,186-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7e89c08f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32f73e0
>2018-05-29 21:50:01,188-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7e89c08f] FINISH, SetVdsStatusVDSCommand, log id: 32f73e0
>2018-05-29 21:50:01,194-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7e89c08f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7e89c08f, Job ID: ad89ad7d-d3a8-41ca-b3ca-eabc1a9d7acb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:50:01,199-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7e89c08f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 15cd92b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:50:01,201-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7e89c08f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 15cd92b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:50:01,211-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [40bdf9ea] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:50:01,212-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [40bdf9ea] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:54:43,320-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler2) [48133d8e] Attempting to update VMs/Templates Ovf.
>2018-05-29 21:54:43,322-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 21:54:43,322-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 21:54:43,323-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:54:43,327-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 21:54:43,331-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 21:54:43,331-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 21:54:43,331-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 21:54:43,331-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 21:54:43,331-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 21:54:43,332-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 21:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [35e023c] Autorecovering 2 hosts
>2018-05-29 21:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [35e023c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:55:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2b07cc64] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:55:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2b07cc64] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:55:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2b07cc64] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:55:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2b07cc64] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:55:00,051-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2b07cc64] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7392f111
>2018-05-29 21:55:00,053-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2b07cc64] FINISH, SetVdsStatusVDSCommand, log id: 7392f111
>2018-05-29 21:55:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2b07cc64] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:55:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2b07cc64] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 21:55:00,056-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [2b07cc64] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:55:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [498ec7ae] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:55:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [498ec7ae] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 21:55:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [498ec7ae] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:55:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [498ec7ae] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:55:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [498ec7ae] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f6c74c6
>2018-05-29 21:55:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [498ec7ae] FINISH, SetVdsStatusVDSCommand, log id: 1f6c74c6
>2018-05-29 21:55:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [498ec7ae] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6293a85e
>2018-05-29 21:55:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [498ec7ae] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6293a85e
>2018-05-29 21:55:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [498ec7ae] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 21:55:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [498ec7ae] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 21:55:01,564-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [52421d76] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: d2ea9e2
>2018-05-29 21:55:01,566-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [52421d76] FINISH, GetHardwareInfoVDSCommand, log id: d2ea9e2
>2018-05-29 21:55:01,579-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [6bce0808] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:55:01,587-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [2d966bbf] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:55:01,592-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [6c8cf6dd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:55:01,598-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [6c8cf6dd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7a7e4415
>2018-05-29 21:55:01,882-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6db8c0ac] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6ed279bc
>2018-05-29 21:55:01,884-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6db8c0ac] FINISH, GetHardwareInfoVDSCommand, log id: 6ed279bc
>2018-05-29 21:55:01,896-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [719093df] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:55:01,901-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [5799d144] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:55:01,907-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [655a668] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 21:55:01,913-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [655a668] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f7badda
>2018-05-29 21:55:01,975-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c8cf6dd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:55:01,975-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [6c8cf6dd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7a7e4415
>2018-05-29 21:55:01,979-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c8cf6dd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:55:01,980-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6c8cf6dd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:55:01,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c8cf6dd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:55:01,984-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6c8cf6dd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:55:01,988-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c8cf6dd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:55:01,989-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6c8cf6dd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:55:01,991-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [6c8cf6dd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:55:01,994-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c8cf6dd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6c8cf6dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:55:02,001-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 45d66a51
>2018-05-29 21:55:02,001-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:55:02,100-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:55:02,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:55:02,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:55:02,100-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:55:02,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 45d66a51
>2018-05-29 21:55:02,100-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:55:02,113-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [5bb48a6e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:55:02,114-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5bb48a6e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45347fc7
>2018-05-29 21:55:02,116-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5bb48a6e] FINISH, SetVdsStatusVDSCommand, log id: 45347fc7
>2018-05-29 21:55:02,147-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bb48a6e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5bb48a6e, Job ID: 445628d2-10be-4e3a-b406-85e0eaa07092, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:55:02,152-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bb48a6e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2d966bbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:55:02,154-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bb48a6e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2d966bbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:55:02,165-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [2975f9db] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 21:55:02,166-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [2975f9db] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 21:55:03,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [655a668] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 21:55:03,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [655a668] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f7badda
>2018-05-29 21:55:03,284-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [655a668] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:55:03,285-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [655a668] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:55:03,289-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [655a668] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:55:03,289-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [655a668] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:55:03,293-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [655a668] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 21:55:03,294-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [655a668] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 21:55:03,297-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [655a668] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 21:55:03,300-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [655a668] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 655a668, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 21:55:03,303-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2de24871
>2018-05-29 21:55:03,303-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 21:55:03,400-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:55:03,400-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 21:55:03,400-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 21:55:03,400-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 21:55:03,400-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2de24871
>2018-05-29 21:55:03,400-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 21:55:03,413-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [2c803754] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:55:03,414-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2c803754] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49ff4554
>2018-05-29 21:55:03,416-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2c803754] FINISH, SetVdsStatusVDSCommand, log id: 49ff4554
>2018-05-29 21:55:03,422-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2c803754] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2c803754, Job ID: 0d8b9f97-8b19-4b23-a48c-71d4cf3b2397, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 21:55:03,426-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2c803754] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5799d144, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 21:55:03,428-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2c803754] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5799d144, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 21:55:03,438-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [43509fe1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 21:55:03,439-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [43509fe1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5fe19c0d] Autorecovering 2 hosts
>2018-05-29 22:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5fe19c0d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:00:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1d551005] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1d551005] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1d551005] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1d551005] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:00:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1d551005] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ef2f1c6
>2018-05-29 22:00:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1d551005] FINISH, SetVdsStatusVDSCommand, log id: 2ef2f1c6
>2018-05-29 22:00:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1d551005] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:00:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1d551005] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:00:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1d551005] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:00:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [58c3d40d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:00:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [58c3d40d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:00:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [58c3d40d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:00:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [58c3d40d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:00:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [58c3d40d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 546b3ab4
>2018-05-29 22:00:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [58c3d40d] FINISH, SetVdsStatusVDSCommand, log id: 546b3ab4
>2018-05-29 22:00:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [58c3d40d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4f2dbe3e
>2018-05-29 22:00:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [58c3d40d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4f2dbe3e
>2018-05-29 22:00:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [58c3d40d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:00:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [58c3d40d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:00:01,127-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4a8ba379] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 266a8c61
>2018-05-29 22:00:01,128-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4a8ba379] FINISH, GetHardwareInfoVDSCommand, log id: 266a8c61
>2018-05-29 22:00:01,142-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [2b183378] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:00:01,146-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [4b41cba8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:00:01,151-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5c6bd7f6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:00:01,163-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5c6bd7f6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 15ea7034
>2018-05-29 22:00:01,534-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c6bd7f6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:00:01,534-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5c6bd7f6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 15ea7034
>2018-05-29 22:00:01,538-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c6bd7f6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:00:01,539-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5c6bd7f6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:00:01,543-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c6bd7f6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:00:01,544-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5c6bd7f6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:00:01,547-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c6bd7f6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:00:01,548-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5c6bd7f6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:00:01,551-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5c6bd7f6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:00:01,553-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c6bd7f6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5c6bd7f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:00:01,560-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4419ec3
>2018-05-29 22:00:01,560-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:00:01,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:00:01,659-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:00:01,659-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:00:01,659-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:00:01,659-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4419ec3
>2018-05-29 22:00:01,659-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:00:01,673-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [548d4b01] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:00:01,674-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [548d4b01] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 74f12d25
>2018-05-29 22:00:01,676-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [548d4b01] FINISH, SetVdsStatusVDSCommand, log id: 74f12d25
>2018-05-29 22:00:01,684-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [548d4b01] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 548d4b01, Job ID: f6d7619f-1a20-4dbc-9973-11e2be15c6de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:00:01,689-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [548d4b01] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4b41cba8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:00:01,691-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [548d4b01] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4b41cba8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:00:01,701-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [3ed29a3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:00:01,702-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [3ed29a3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:00:02,903-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [142c523f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3e937e99
>2018-05-29 22:00:02,904-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [142c523f] FINISH, GetHardwareInfoVDSCommand, log id: 3e937e99
>2018-05-29 22:00:02,918-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [20296a1e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:00:02,922-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [3194d285] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:00:02,927-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [44472301] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:00:02,934-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [44472301] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7140928f
>2018-05-29 22:00:03,311-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44472301] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:00:03,311-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [44472301] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7140928f
>2018-05-29 22:00:03,314-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44472301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:00:03,315-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [44472301] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:00:03,319-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44472301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:00:03,320-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [44472301] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:00:03,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44472301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:00:03,324-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [44472301] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:00:03,327-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [44472301] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:00:03,330-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44472301] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 44472301, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:00:03,334-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 20bf9e50
>2018-05-29 22:00:03,334-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:00:03,429-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:00:03,429-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:00:03,430-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:00:03,430-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:00:03,430-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 20bf9e50
>2018-05-29 22:00:03,430-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:00:03,442-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [89f2323] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:00:03,444-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [89f2323] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41cf5651
>2018-05-29 22:00:03,446-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [89f2323] FINISH, SetVdsStatusVDSCommand, log id: 41cf5651
>2018-05-29 22:00:03,451-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [89f2323] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 89f2323, Job ID: 301409aa-4c33-4e92-97fa-495f6adb33f7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:00:03,456-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [89f2323] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3194d285, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:00:03,458-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [89f2323] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3194d285, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:00:03,468-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [67a5edd6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:00:03,469-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [67a5edd6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [156d4a8a] Autorecovering 2 hosts
>2018-05-29 22:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [156d4a8a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:05:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7f90b4f2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:05:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7f90b4f2] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:05:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7f90b4f2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:05:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7f90b4f2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:05:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7f90b4f2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 71ef797d
>2018-05-29 22:05:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7f90b4f2] FINISH, SetVdsStatusVDSCommand, log id: 71ef797d
>2018-05-29 22:05:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7f90b4f2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:05:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7f90b4f2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:05:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7f90b4f2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:05:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [528904f9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:05:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [528904f9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:05:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [528904f9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:05:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [528904f9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:05:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [528904f9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6404265e
>2018-05-29 22:05:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [528904f9] FINISH, SetVdsStatusVDSCommand, log id: 6404265e
>2018-05-29 22:05:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [528904f9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 346ab594
>2018-05-29 22:05:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [528904f9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 346ab594
>2018-05-29 22:05:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [528904f9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:05:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [528904f9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:05:01,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [16a5e932] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 33f7b4bd
>2018-05-29 22:05:01,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [16a5e932] FINISH, GetHardwareInfoVDSCommand, log id: 33f7b4bd
>2018-05-29 22:05:01,115-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [4f41b8ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:05:01,124-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [77882535] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:05:01,129-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [82aef05] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:05:01,140-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [82aef05] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 75ea3437
>2018-05-29 22:05:01,528-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [82aef05] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:05:01,528-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [82aef05] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 75ea3437
>2018-05-29 22:05:01,532-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [82aef05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:05:01,533-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [82aef05] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:05:01,537-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [82aef05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:05:01,538-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [82aef05] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:05:01,541-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [82aef05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:05:01,542-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [82aef05] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:05:01,544-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [82aef05] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:05:01,547-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [82aef05] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 82aef05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:05:01,553-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 171132a
>2018-05-29 22:05:01,553-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:05:01,652-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:05:01,652-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:05:01,652-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:05:01,652-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:05:01,652-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 171132a
>2018-05-29 22:05:01,652-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:05:01,666-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [22c543f6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:05:01,667-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [22c543f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 296543d7
>2018-05-29 22:05:01,670-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [22c543f6] FINISH, SetVdsStatusVDSCommand, log id: 296543d7
>2018-05-29 22:05:01,702-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [22c543f6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22c543f6, Job ID: f755f6bb-600f-4ebf-8d5f-de3192c8fe9a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:05:01,707-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [22c543f6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 77882535, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:05:01,709-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [22c543f6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 77882535, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:05:01,720-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [25ede32] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:05:01,721-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [25ede32] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:05:02,465-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [29dc36ec] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5b678ed6
>2018-05-29 22:05:02,467-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [29dc36ec] FINISH, GetHardwareInfoVDSCommand, log id: 5b678ed6
>2018-05-29 22:05:02,480-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [da2764f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:05:02,485-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [2604d6ff] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:05:02,490-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [20d9ce58] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:05:02,497-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [20d9ce58] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5b4df2ae
>2018-05-29 22:05:02,861-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [20d9ce58] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:05:02,861-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [20d9ce58] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5b4df2ae
>2018-05-29 22:05:02,865-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [20d9ce58] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:05:02,865-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [20d9ce58] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:05:02,869-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [20d9ce58] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:05:02,870-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [20d9ce58] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:05:02,873-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [20d9ce58] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:05:02,874-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [20d9ce58] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:05:02,877-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [20d9ce58] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:05:02,879-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [20d9ce58] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 20d9ce58, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:05:02,883-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 45de65ac
>2018-05-29 22:05:02,883-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:05:02,979-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:05:02,979-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:05:02,979-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:05:02,979-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:05:02,979-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 45de65ac
>2018-05-29 22:05:02,979-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:05:02,991-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [5dfddcbc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:05:02,992-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5dfddcbc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50aeb859
>2018-05-29 22:05:02,994-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5dfddcbc] FINISH, SetVdsStatusVDSCommand, log id: 50aeb859
>2018-05-29 22:05:02,999-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5dfddcbc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5dfddcbc, Job ID: f8389a33-14fd-481d-9922-747944862cf8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:05:03,004-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5dfddcbc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2604d6ff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:05:03,006-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5dfddcbc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2604d6ff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:05:03,016-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [47246590] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:05:03,017-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [47246590] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [ba6a01c] Autorecovering 2 hosts
>2018-05-29 22:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [ba6a01c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:10:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3860f186] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:10:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3860f186] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:10:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3860f186] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:10:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3860f186] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:10:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3860f186] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42f00e8b
>2018-05-29 22:10:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3860f186] FINISH, SetVdsStatusVDSCommand, log id: 42f00e8b
>2018-05-29 22:10:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3860f186] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:10:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3860f186] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:10:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3860f186] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:10:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cf44ad0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:10:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cf44ad0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:10:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cf44ad0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:10:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cf44ad0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:10:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2cf44ad0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1251800f
>2018-05-29 22:10:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2cf44ad0] FINISH, SetVdsStatusVDSCommand, log id: 1251800f
>2018-05-29 22:10:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [2cf44ad0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 73ebfbc7
>2018-05-29 22:10:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [2cf44ad0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 73ebfbc7
>2018-05-29 22:10:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cf44ad0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:10:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cf44ad0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:10:00,680-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [79e1c6de] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: eca7720
>2018-05-29 22:10:00,682-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [79e1c6de] FINISH, GetHardwareInfoVDSCommand, log id: eca7720
>2018-05-29 22:10:00,694-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [64bebb96] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:10:00,702-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [6d6fabfa] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:10:00,707-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51519e1a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:10:00,719-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51519e1a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 38561538
>2018-05-29 22:10:01,095-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51519e1a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:10:01,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51519e1a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 38561538
>2018-05-29 22:10:01,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51519e1a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:10:01,100-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51519e1a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:10:01,104-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51519e1a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:10:01,104-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51519e1a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:10:01,108-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51519e1a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:10:01,109-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51519e1a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:10:01,112-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51519e1a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:10:01,115-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51519e1a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 51519e1a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:10:01,118-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 854c251
>2018-05-29 22:10:01,119-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:10:01,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:10:01,218-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:10:01,218-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:10:01,218-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:10:01,218-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 854c251
>2018-05-29 22:10:01,218-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:10:01,231-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [29b35015] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:10:01,233-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [29b35015] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 333bb7b
>2018-05-29 22:10:01,235-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [29b35015] FINISH, SetVdsStatusVDSCommand, log id: 333bb7b
>2018-05-29 22:10:01,265-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [29b35015] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 29b35015, Job ID: 56185d3c-6075-4d31-8e6e-7f030c763e01, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:10:01,270-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [29b35015] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6d6fabfa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:10:01,272-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [29b35015] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6d6fabfa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:10:01,282-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [63d58cb2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:10:01,284-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [63d58cb2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:10:02,344-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [24e38675] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 11049ee9
>2018-05-29 22:10:02,346-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [24e38675] FINISH, GetHardwareInfoVDSCommand, log id: 11049ee9
>2018-05-29 22:10:02,358-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [c0d14b9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:10:02,366-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [49732b8d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:10:02,371-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [71d2fcbe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:10:02,377-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [71d2fcbe] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 158dc11d
>2018-05-29 22:10:02,749-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [71d2fcbe] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:10:02,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [71d2fcbe] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 158dc11d
>2018-05-29 22:10:02,752-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [71d2fcbe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:10:02,753-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [71d2fcbe] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:10:02,757-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [71d2fcbe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:10:02,757-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [71d2fcbe] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:10:02,761-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [71d2fcbe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:10:02,762-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [71d2fcbe] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:10:02,764-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [71d2fcbe] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:10:02,767-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [71d2fcbe] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 71d2fcbe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:10:02,769-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 34cacbd8
>2018-05-29 22:10:02,769-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:10:02,865-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:10:02,865-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:10:02,865-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:10:02,865-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:10:02,865-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 34cacbd8
>2018-05-29 22:10:02,865-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:10:02,878-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [29505c50] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:10:02,879-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [29505c50] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 21a24fbf
>2018-05-29 22:10:02,881-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [29505c50] FINISH, SetVdsStatusVDSCommand, log id: 21a24fbf
>2018-05-29 22:10:02,887-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [29505c50] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 29505c50, Job ID: eb927b01-8dee-4d1e-81f2-17bb8939bfc9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:10:02,891-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [29505c50] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 49732b8d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:10:02,893-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [29505c50] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 49732b8d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:10:02,904-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5505e11c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:10:02,905-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5505e11c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [1e91abfb] Autorecovering 2 hosts
>2018-05-29 22:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [1e91abfb] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:15:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [116c8874] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [116c8874] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [116c8874] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [116c8874] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:15:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [116c8874] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 223e3204
>2018-05-29 22:15:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [116c8874] FINISH, SetVdsStatusVDSCommand, log id: 223e3204
>2018-05-29 22:15:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [116c8874] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:15:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [116c8874] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:15:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [116c8874] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:15:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1cc56030] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:15:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1cc56030] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:15:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1cc56030] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:15:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1cc56030] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:15:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1cc56030] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4a95403
>2018-05-29 22:15:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1cc56030] FINISH, SetVdsStatusVDSCommand, log id: 4a95403
>2018-05-29 22:15:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [1cc56030] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1fb60029
>2018-05-29 22:15:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [1cc56030] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1fb60029
>2018-05-29 22:15:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1cc56030] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:15:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1cc56030] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:15:00,538-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [13859967] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 53ba3c72
>2018-05-29 22:15:00,540-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [13859967] FINISH, GetHardwareInfoVDSCommand, log id: 53ba3c72
>2018-05-29 22:15:00,553-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [2df48586] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:15:00,558-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [17e0d6a2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:15:00,563-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [68f5eb08] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:15:00,620-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [68f5eb08] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 69727c16
>2018-05-29 22:15:01,004-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [68f5eb08] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:15:01,004-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [68f5eb08] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 69727c16
>2018-05-29 22:15:01,008-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [68f5eb08] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:15:01,009-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [68f5eb08] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:15:01,013-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [68f5eb08] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:15:01,014-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [68f5eb08] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:15:01,018-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [68f5eb08] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:15:01,019-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [68f5eb08] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:15:01,021-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [68f5eb08] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:15:01,024-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [68f5eb08] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 68f5eb08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:15:01,031-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6b45ed25
>2018-05-29 22:15:01,031-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:15:01,128-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:15:01,128-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:15:01,128-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:15:01,128-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:15:01,128-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6b45ed25
>2018-05-29 22:15:01,128-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:15:01,142-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [3f0252ef] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:15:01,143-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3f0252ef] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4698ed6c
>2018-05-29 22:15:01,146-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3f0252ef] FINISH, SetVdsStatusVDSCommand, log id: 4698ed6c
>2018-05-29 22:15:01,179-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f0252ef] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3f0252ef, Job ID: b8995fd8-2105-419c-97ca-f7928377cf02, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:15:01,183-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f0252ef] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17e0d6a2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:15:01,186-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f0252ef] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17e0d6a2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:15:01,197-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6bee2244] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:15:01,198-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6bee2244] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:15:01,909-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [2a7e1e97] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 672a43a2
>2018-05-29 22:15:01,911-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [2a7e1e97] FINISH, GetHardwareInfoVDSCommand, log id: 672a43a2
>2018-05-29 22:15:01,923-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [2e6bdf08] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:15:01,927-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [641255dd] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:15:01,932-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [41043288] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:15:01,945-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [41043288] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 53522a74
>2018-05-29 22:15:03,319-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41043288] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:15:03,319-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [41043288] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 53522a74
>2018-05-29 22:15:03,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41043288] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:15:03,324-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [41043288] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:15:03,328-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41043288] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:15:03,329-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [41043288] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:15:03,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41043288] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:15:03,333-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [41043288] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:15:03,336-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [41043288] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:15:03,339-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41043288] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 41043288, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:15:03,346-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: d4bb3b
>2018-05-29 22:15:03,346-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:15:03,442-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:15:03,442-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:15:03,442-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:15:03,442-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:15:03,442-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: d4bb3b
>2018-05-29 22:15:03,442-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:15:03,455-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [22350e22] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:15:03,456-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [22350e22] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: eddf066
>2018-05-29 22:15:03,458-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [22350e22] FINISH, SetVdsStatusVDSCommand, log id: eddf066
>2018-05-29 22:15:03,463-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [22350e22] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22350e22, Job ID: 1538104f-e673-40f2-8de3-790b23ce75c0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:15:03,467-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [22350e22] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 641255dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:15:03,469-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [22350e22] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 641255dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:15:03,480-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [72ea9069] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:15:03,481-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [72ea9069] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:20:00,016-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1c7a5020] Autorecovering 2 hosts
>2018-05-29 22:20:00,016-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1c7a5020] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:20:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c39371c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:20:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c39371c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:20:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c39371c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:20:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c39371c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:20:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6c39371c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 309d3955
>2018-05-29 22:20:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6c39371c] FINISH, SetVdsStatusVDSCommand, log id: 309d3955
>2018-05-29 22:20:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c39371c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:20:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c39371c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:20:00,064-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6c39371c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:20:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7177353a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:20:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7177353a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:20:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7177353a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:20:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7177353a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:20:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7177353a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 64dc81d9
>2018-05-29 22:20:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7177353a] FINISH, SetVdsStatusVDSCommand, log id: 64dc81d9
>2018-05-29 22:20:00,081-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7177353a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1c302077
>2018-05-29 22:20:00,084-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7177353a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1c302077
>2018-05-29 22:20:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7177353a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:20:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7177353a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:20:01,198-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [3e914c83] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 782b2574
>2018-05-29 22:20:01,200-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [3e914c83] FINISH, GetHardwareInfoVDSCommand, log id: 782b2574
>2018-05-29 22:20:01,213-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [6c61432] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:20:01,221-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [76a38157] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:20:01,226-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [37a90c24] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:20:01,232-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [37a90c24] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2244b826
>2018-05-29 22:20:01,605-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [37a90c24] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:20:01,605-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [37a90c24] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2244b826
>2018-05-29 22:20:01,609-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [37a90c24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:20:01,610-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [37a90c24] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:20:01,613-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [37a90c24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:20:01,614-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [37a90c24] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:20:01,618-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [37a90c24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:20:01,618-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [37a90c24] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:20:01,621-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [37a90c24] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:20:01,624-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [37a90c24] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 37a90c24, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:20:01,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 67435aef
>2018-05-29 22:20:01,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:20:01,727-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:20:01,727-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:20:01,727-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:20:01,727-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:20:01,727-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: 67435aef
>2018-05-29 22:20:01,727-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:20:01,739-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [4a0cae08] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:20:01,740-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4a0cae08] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 231a9d50
>2018-05-29 22:20:01,743-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4a0cae08] FINISH, SetVdsStatusVDSCommand, log id: 231a9d50
>2018-05-29 22:20:01,773-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4a0cae08] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4a0cae08, Job ID: bf574c7a-98b6-4edd-a907-d10ebbcfe3fe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:20:01,777-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4a0cae08] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 76a38157, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:20:01,779-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4a0cae08] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 76a38157, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:20:01,790-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [2dc50a6e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:20:01,791-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [2dc50a6e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:20:01,847-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [125db901] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7830d29a
>2018-05-29 22:20:01,848-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [125db901] FINISH, GetHardwareInfoVDSCommand, log id: 7830d29a
>2018-05-29 22:20:01,860-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [3a156e12] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:20:01,864-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [6a0c1de0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:20:01,870-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [54f5a9a1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:20:01,876-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [54f5a9a1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 55841a95
>2018-05-29 22:20:02,251-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54f5a9a1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:20:02,251-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [54f5a9a1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 55841a95
>2018-05-29 22:20:02,255-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54f5a9a1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:20:02,255-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [54f5a9a1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:20:02,259-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54f5a9a1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:20:02,260-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [54f5a9a1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:20:02,264-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54f5a9a1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:20:02,264-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [54f5a9a1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:20:02,267-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [54f5a9a1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:20:02,270-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54f5a9a1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 54f5a9a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:20:02,273-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5139ad8e
>2018-05-29 22:20:02,273-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:20:02,370-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:20:02,371-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:20:02,371-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:20:02,371-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:20:02,371-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5139ad8e
>2018-05-29 22:20:02,371-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:20:02,383-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [1334538a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:20:02,385-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1334538a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 78d9f2e5
>2018-05-29 22:20:02,387-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1334538a] FINISH, SetVdsStatusVDSCommand, log id: 78d9f2e5
>2018-05-29 22:20:02,392-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1334538a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1334538a, Job ID: 9e26fb77-ccfd-45d2-8b75-a98ca5641007, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:20:02,397-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1334538a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6a0c1de0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:20:02,399-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1334538a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6a0c1de0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:20:02,409-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [555be83] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:20:02,410-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [555be83] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5cc79e73] Autorecovering 2 hosts
>2018-05-29 22:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5cc79e73] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:25:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [63961d74] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:25:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [63961d74] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:25:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [63961d74] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:25:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [63961d74] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:25:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [63961d74] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 36f80713
>2018-05-29 22:25:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [63961d74] FINISH, SetVdsStatusVDSCommand, log id: 36f80713
>2018-05-29 22:25:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [63961d74] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:25:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [63961d74] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:25:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [63961d74] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:25:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1c4de061] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:25:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1c4de061] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:25:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1c4de061] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:25:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1c4de061] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:25:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1c4de061] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42502daa
>2018-05-29 22:25:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1c4de061] FINISH, SetVdsStatusVDSCommand, log id: 42502daa
>2018-05-29 22:25:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [1c4de061] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5ad53a32
>2018-05-29 22:25:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [1c4de061] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5ad53a32
>2018-05-29 22:25:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1c4de061] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:25:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1c4de061] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:25:02,571-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [679a19ae] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 56b707ed
>2018-05-29 22:25:02,573-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [679a19ae] FINISH, GetHardwareInfoVDSCommand, log id: 56b707ed
>2018-05-29 22:25:02,588-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2d7075f4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:25:02,597-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4a8787e3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:25:02,602-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1977ad99] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:25:02,608-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1977ad99] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 57b9362f
>2018-05-29 22:25:03,003-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5543f98a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 647cb06e
>2018-05-29 22:25:03,004-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5543f98a] FINISH, GetHardwareInfoVDSCommand, log id: 647cb06e
>2018-05-29 22:25:03,016-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [7b3ab04c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:25:03,021-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [34bb238e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:25:03,026-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [89ef2b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:25:03,032-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [89ef2b8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 47173f07
>2018-05-29 22:25:03,402-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [89ef2b8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:25:03,402-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [89ef2b8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 47173f07
>2018-05-29 22:25:03,406-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [89ef2b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:25:03,407-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [89ef2b8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:25:03,411-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [89ef2b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:25:03,412-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [89ef2b8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:25:03,416-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [89ef2b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:25:03,416-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [89ef2b8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:25:03,419-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [89ef2b8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:25:03,422-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [89ef2b8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 89ef2b8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:25:03,425-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6adc4b1
>2018-05-29 22:25:03,425-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:25:03,518-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:25:03,519-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:25:03,519-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:25:03,519-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:25:03,519-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6adc4b1
>2018-05-29 22:25:03,519-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:25:03,532-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [7eb65b56] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:25:03,533-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7eb65b56] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26988c09
>2018-05-29 22:25:03,535-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7eb65b56] FINISH, SetVdsStatusVDSCommand, log id: 26988c09
>2018-05-29 22:25:03,567-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7eb65b56] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7eb65b56, Job ID: 2faecb16-9b99-4cfa-90c6-5c541e376ba8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:25:03,571-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7eb65b56] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 34bb238e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:25:03,573-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7eb65b56] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 34bb238e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:25:03,584-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [383c0318] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:25:03,585-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [383c0318] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:25:03,976-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1977ad99] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:25:03,976-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1977ad99] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 57b9362f
>2018-05-29 22:25:03,980-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1977ad99] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:25:03,980-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1977ad99] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:25:03,984-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1977ad99] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:25:03,985-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1977ad99] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:25:03,989-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1977ad99] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:25:03,990-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1977ad99] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:25:03,994-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1977ad99] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:25:03,997-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1977ad99] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1977ad99, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:25:04,000-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1048769b
>2018-05-29 22:25:04,000-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:25:04,095-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:25:04,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:25:04,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:25:04,095-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:25:04,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1048769b
>2018-05-29 22:25:04,095-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:25:04,109-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [20b024e5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:25:04,110-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [20b024e5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6980e7ee
>2018-05-29 22:25:04,113-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [20b024e5] FINISH, SetVdsStatusVDSCommand, log id: 6980e7ee
>2018-05-29 22:25:04,118-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [20b024e5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 20b024e5, Job ID: c22dc5e2-699d-4239-998a-7f9fc84cac06, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:25:04,122-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [20b024e5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4a8787e3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:25:04,124-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [20b024e5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4a8787e3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:25:04,135-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [65d972be] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:25:04,136-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [65d972be] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6fbaf3b6] Autorecovering 2 hosts
>2018-05-29 22:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6fbaf3b6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:30:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1b302f85] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:30:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1b302f85] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:30:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1b302f85] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:30:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1b302f85] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:30:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1b302f85] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30c7c69e
>2018-05-29 22:30:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1b302f85] FINISH, SetVdsStatusVDSCommand, log id: 30c7c69e
>2018-05-29 22:30:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1b302f85] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:30:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1b302f85] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:30:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1b302f85] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:30:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7e05fe] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:30:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7e05fe] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:30:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7e05fe] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:30:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7e05fe] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:30:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7e05fe] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32c7799e
>2018-05-29 22:30:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7e05fe] FINISH, SetVdsStatusVDSCommand, log id: 32c7799e
>2018-05-29 22:30:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7e05fe] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 563baff3
>2018-05-29 22:30:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7e05fe] FINISH, SetHaMaintenanceModeVDSCommand, log id: 563baff3
>2018-05-29 22:30:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7e05fe] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:30:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7e05fe] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:30:01,268-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4496f797] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 50c33d69
>2018-05-29 22:30:01,270-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4496f797] FINISH, GetHardwareInfoVDSCommand, log id: 50c33d69
>2018-05-29 22:30:01,283-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [7fb8cc94] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:30:01,291-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [734c2566] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:30:01,296-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4e251905] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:30:01,308-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4e251905] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 246ebacd
>2018-05-29 22:30:01,691-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e251905] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:30:01,691-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4e251905] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 246ebacd
>2018-05-29 22:30:01,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e251905] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:30:01,695-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4e251905] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:30:01,699-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e251905] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:30:01,700-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4e251905] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:30:01,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e251905] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:30:01,705-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4e251905] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:30:01,708-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4e251905] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:30:01,711-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e251905] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4e251905, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:30:01,718-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3cc8b204
>2018-05-29 22:30:01,718-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:30:01,766-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5cbbf6a6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3f707ee0
>2018-05-29 22:30:01,768-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5cbbf6a6] FINISH, GetHardwareInfoVDSCommand, log id: 3f707ee0
>2018-05-29 22:30:01,780-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [445b7c1f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:30:01,789-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [18f00fff] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:30:01,794-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [668d6ace] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:30:01,806-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [668d6ace] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 663d7517
>2018-05-29 22:30:01,817-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:30:01,817-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:30:01,817-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:30:01,817-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:30:01,817-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3cc8b204
>2018-05-29 22:30:01,817-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:30:01,830-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [20d43faa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:30:01,831-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [20d43faa] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ea44f86
>2018-05-29 22:30:01,834-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [20d43faa] FINISH, SetVdsStatusVDSCommand, log id: 3ea44f86
>2018-05-29 22:30:01,866-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [20d43faa] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 20d43faa, Job ID: 14ec65de-c20d-48ca-aac6-83dcdaaecfd8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:30:01,872-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [20d43faa] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 734c2566, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:30:01,874-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [20d43faa] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 734c2566, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:30:01,885-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [1f59ac02] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:30:01,886-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [1f59ac02] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:30:02,180-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [668d6ace] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:30:02,180-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [668d6ace] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 663d7517
>2018-05-29 22:30:02,183-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [668d6ace] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:30:02,184-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [668d6ace] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:30:02,188-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [668d6ace] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:30:02,189-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [668d6ace] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:30:02,193-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [668d6ace] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:30:02,194-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [668d6ace] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:30:02,196-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [668d6ace] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:30:02,199-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [668d6ace] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 668d6ace, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:30:02,202-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4bbd737e
>2018-05-29 22:30:02,202-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:30:02,301-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:30:02,301-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:30:02,301-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:30:02,301-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:30:02,301-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4bbd737e
>2018-05-29 22:30:02,301-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:30:02,314-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [451c89ac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:30:02,315-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [451c89ac] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 56263232
>2018-05-29 22:30:02,317-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [451c89ac] FINISH, SetVdsStatusVDSCommand, log id: 56263232
>2018-05-29 22:30:02,322-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [451c89ac] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 451c89ac, Job ID: 16638329-e209-4961-b0b0-3f13c15d054e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:30:02,327-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [451c89ac] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 18f00fff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:30:02,329-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [451c89ac] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 18f00fff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:30:02,340-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [6aa1e7ac] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:30:02,341-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [6aa1e7ac] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:35:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [6e8ac1d6] Autorecovering 2 hosts
>2018-05-29 22:35:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [6e8ac1d6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:35:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3003f38a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:35:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3003f38a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:35:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3003f38a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:35:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3003f38a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:35:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3003f38a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 22912f58
>2018-05-29 22:35:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3003f38a] FINISH, SetVdsStatusVDSCommand, log id: 22912f58
>2018-05-29 22:35:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3003f38a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:35:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3003f38a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:35:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [3003f38a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:35:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [794732e7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:35:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [794732e7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:35:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [794732e7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:35:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [794732e7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:35:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [794732e7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3681ee13
>2018-05-29 22:35:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [794732e7] FINISH, SetVdsStatusVDSCommand, log id: 3681ee13
>2018-05-29 22:35:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [794732e7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5067b7b7
>2018-05-29 22:35:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [794732e7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5067b7b7
>2018-05-29 22:35:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [794732e7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:35:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [794732e7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:35:02,510-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [770ca8a2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 602d02a7
>2018-05-29 22:35:02,512-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [770ca8a2] FINISH, GetHardwareInfoVDSCommand, log id: 602d02a7
>2018-05-29 22:35:02,527-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [52b9e49a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:35:02,536-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [312e40d3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:35:02,542-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1df8bc39] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:35:02,549-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1df8bc39] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 799cc3f9
>2018-05-29 22:35:02,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1df8bc39] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:35:02,935-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1df8bc39] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 799cc3f9
>2018-05-29 22:35:02,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1df8bc39] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:35:02,939-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1df8bc39] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:35:02,943-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1df8bc39] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:35:02,944-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1df8bc39] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:35:02,948-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1df8bc39] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:35:02,948-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1df8bc39] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:35:02,951-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1df8bc39] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:35:02,954-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1df8bc39] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1df8bc39, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:35:02,957-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4777f285
>2018-05-29 22:35:02,957-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:35:03,004-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [62927ea3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1009adae
>2018-05-29 22:35:03,006-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [62927ea3] FINISH, GetHardwareInfoVDSCommand, log id: 1009adae
>2018-05-29 22:35:03,019-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [695a00a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:35:03,023-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [31cf9b54] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:35:03,028-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [63bcfca3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:35:03,034-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [63bcfca3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: cb5c05b
>2018-05-29 22:35:03,055-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:35:03,055-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:35:03,055-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:35:03,055-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:35:03,055-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4777f285
>2018-05-29 22:35:03,055-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:35:03,069-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [aa5e1f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:35:03,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [aa5e1f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 55f5a843
>2018-05-29 22:35:03,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [aa5e1f] FINISH, SetVdsStatusVDSCommand, log id: 55f5a843
>2018-05-29 22:35:03,103-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [aa5e1f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: aa5e1f, Job ID: 2b13c5a6-7f60-4f24-b660-bb4e3ac9e5ae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:35:03,108-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [aa5e1f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 312e40d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:35:03,110-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [aa5e1f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 312e40d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:35:03,121-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [30871601] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:35:03,122-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [30871601] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:35:04,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [63bcfca3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:35:04,401-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [63bcfca3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: cb5c05b
>2018-05-29 22:35:04,405-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [63bcfca3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:35:04,406-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [63bcfca3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:35:04,409-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [63bcfca3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:35:04,410-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [63bcfca3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:35:04,414-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [63bcfca3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:35:04,415-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [63bcfca3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:35:04,418-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [63bcfca3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:35:04,421-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [63bcfca3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 63bcfca3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:35:04,424-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 288f05df
>2018-05-29 22:35:04,424-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:35:04,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:35:04,520-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:35:04,520-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:35:04,520-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:35:04,520-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 288f05df
>2018-05-29 22:35:04,520-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:35:04,533-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [65a60336] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:35:04,534-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [65a60336] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 273fce1
>2018-05-29 22:35:04,537-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [65a60336] FINISH, SetVdsStatusVDSCommand, log id: 273fce1
>2018-05-29 22:35:04,543-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [65a60336] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 65a60336, Job ID: 244c1435-de5e-49b7-803d-9905fe9fbf47, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:35:04,547-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [65a60336] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 31cf9b54, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:35:04,550-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [65a60336] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 31cf9b54, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:35:04,560-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [44b73d75] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:35:04,561-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [44b73d75] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:40:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [54e7535a] Autorecovering 2 hosts
>2018-05-29 22:40:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [54e7535a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:40:00,032-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1a43452d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:40:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1a43452d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:40:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1a43452d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:40:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1a43452d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:40:00,040-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1a43452d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b7463d9
>2018-05-29 22:40:00,043-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1a43452d] FINISH, SetVdsStatusVDSCommand, log id: b7463d9
>2018-05-29 22:40:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1a43452d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:40:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1a43452d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:40:00,045-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1a43452d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:40:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [cff14f0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:40:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [cff14f0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:40:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [cff14f0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:40:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [cff14f0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:40:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [cff14f0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 629ff9
>2018-05-29 22:40:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [cff14f0] FINISH, SetVdsStatusVDSCommand, log id: 629ff9
>2018-05-29 22:40:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [cff14f0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6756b58c
>2018-05-29 22:40:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [cff14f0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6756b58c
>2018-05-29 22:40:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [cff14f0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:40:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [cff14f0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:40:00,712-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2ed49ebe] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6dfb4b34
>2018-05-29 22:40:00,714-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2ed49ebe] FINISH, GetHardwareInfoVDSCommand, log id: 6dfb4b34
>2018-05-29 22:40:00,727-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [326dd0b0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:40:00,736-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [3b985519] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:40:00,741-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1ebb799a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:40:00,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1ebb799a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 85de922
>2018-05-29 22:40:01,134-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ebb799a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:40:01,134-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1ebb799a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 85de922
>2018-05-29 22:40:01,138-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ebb799a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:40:01,138-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1ebb799a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:40:01,142-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ebb799a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:40:01,143-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1ebb799a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:40:01,147-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ebb799a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:40:01,148-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1ebb799a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:40:01,151-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1ebb799a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:40:01,154-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ebb799a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1ebb799a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:40:01,160-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 215c9aaf
>2018-05-29 22:40:01,160-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:40:01,254-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:40:01,254-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:40:01,254-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:40:01,254-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:40:01,254-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 215c9aaf
>2018-05-29 22:40:01,254-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:40:01,267-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [1311c231] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:40:01,269-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1311c231] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 322fa388
>2018-05-29 22:40:01,271-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1311c231] FINISH, SetVdsStatusVDSCommand, log id: 322fa388
>2018-05-29 22:40:01,279-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1311c231] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1311c231, Job ID: 0fb017ce-3964-4a65-b717-60885dc6e671, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:40:01,284-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1311c231] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b985519, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:40:01,286-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1311c231] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b985519, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:40:01,297-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5ff43d28] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:40:01,298-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5ff43d28] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:40:02,253-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [56fb568b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1fe1817a
>2018-05-29 22:40:02,255-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [56fb568b] FINISH, GetHardwareInfoVDSCommand, log id: 1fe1817a
>2018-05-29 22:40:02,269-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [7a17b567] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:40:02,273-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [34f88989] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:40:02,278-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [63f01] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:40:02,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [63f01] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 16fc8a6b
>2018-05-29 22:40:02,657-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63f01] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:40:02,657-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [63f01] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 16fc8a6b
>2018-05-29 22:40:02,661-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63f01] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:40:02,662-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [63f01] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:40:02,666-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63f01] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:40:02,666-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [63f01] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:40:02,670-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63f01] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:40:02,671-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [63f01] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:40:02,673-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [63f01] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:40:02,676-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63f01] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 63f01, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:40:02,679-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1463b09e
>2018-05-29 22:40:02,679-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:40:02,782-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:40:02,783-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:40:02,783-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:40:02,783-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:40:02,783-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1463b09e
>2018-05-29 22:40:02,783-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:40:02,796-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [6f31a48a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:40:02,797-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6f31a48a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1235915d
>2018-05-29 22:40:02,800-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6f31a48a] FINISH, SetVdsStatusVDSCommand, log id: 1235915d
>2018-05-29 22:40:02,831-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6f31a48a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6f31a48a, Job ID: 11b1ce51-f8d8-4597-8c63-00b2b75c5631, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:40:02,836-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6f31a48a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 34f88989, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:40:02,838-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6f31a48a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 34f88989, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:40:02,849-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [686849e6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:40:02,850-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [686849e6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1780028e] Autorecovering 2 hosts
>2018-05-29 22:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1780028e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:45:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [977435d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:45:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [977435d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:45:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [977435d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:45:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [977435d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:45:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [977435d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7cf1adef
>2018-05-29 22:45:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [977435d] FINISH, SetVdsStatusVDSCommand, log id: 7cf1adef
>2018-05-29 22:45:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [977435d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:45:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [977435d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:45:00,073-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [977435d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:45:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5e0e8b65] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:45:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5e0e8b65] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:45:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5e0e8b65] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:45:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5e0e8b65] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:45:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5e0e8b65] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7764e436
>2018-05-29 22:45:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5e0e8b65] FINISH, SetVdsStatusVDSCommand, log id: 7764e436
>2018-05-29 22:45:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5e0e8b65] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 158d3fc9
>2018-05-29 22:45:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5e0e8b65] FINISH, SetHaMaintenanceModeVDSCommand, log id: 158d3fc9
>2018-05-29 22:45:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5e0e8b65] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:45:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5e0e8b65] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:45:00,704-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7ee6acfa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3baed269
>2018-05-29 22:45:00,706-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7ee6acfa] FINISH, GetHardwareInfoVDSCommand, log id: 3baed269
>2018-05-29 22:45:00,719-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [42f67f39] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:45:00,727-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [29b31a80] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:45:00,732-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [520d088e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:45:00,744-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [520d088e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4b3e0d6a
>2018-05-29 22:45:01,116-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [520d088e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:45:01,116-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [520d088e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4b3e0d6a
>2018-05-29 22:45:01,121-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [520d088e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:45:01,121-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [520d088e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:45:01,125-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [520d088e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:45:01,126-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [520d088e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:45:01,130-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [520d088e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:45:01,131-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [520d088e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:45:01,134-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [520d088e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:45:01,137-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [520d088e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 520d088e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:45:01,144-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 61f36a85
>2018-05-29 22:45:01,144-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:45:01,242-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:45:01,242-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:45:01,242-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:45:01,242-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:45:01,242-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 61f36a85
>2018-05-29 22:45:01,242-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:45:01,256-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [6a8cdc55] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:45:01,257-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6a8cdc55] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5c0bb236
>2018-05-29 22:45:01,259-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6a8cdc55] FINISH, SetVdsStatusVDSCommand, log id: 5c0bb236
>2018-05-29 22:45:01,290-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6a8cdc55] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6a8cdc55, Job ID: d56dbd96-7298-4a0c-a4e4-8bf8087abc2d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:45:01,295-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6a8cdc55] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29b31a80, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:45:01,297-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6a8cdc55] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29b31a80, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:45:01,308-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [33940d7d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:45:01,309-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [33940d7d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:45:02,176-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6cd8f557] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7df1962a
>2018-05-29 22:45:02,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6cd8f557] FINISH, GetHardwareInfoVDSCommand, log id: 7df1962a
>2018-05-29 22:45:02,192-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [3c568350] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:45:02,196-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [442cb1f1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:45:02,201-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [74e910a1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:45:02,208-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [74e910a1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 47093e3
>2018-05-29 22:45:02,587-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74e910a1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:45:02,587-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [74e910a1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 47093e3
>2018-05-29 22:45:02,591-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74e910a1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:45:02,592-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [74e910a1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:45:02,596-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74e910a1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:45:02,597-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [74e910a1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:45:02,600-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74e910a1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:45:02,601-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [74e910a1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:45:02,604-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [74e910a1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:45:02,606-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74e910a1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 74e910a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:45:02,609-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 21e8ad5a
>2018-05-29 22:45:02,609-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:45:02,705-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:45:02,705-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:45:02,705-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:45:02,705-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:45:02,705-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 21e8ad5a
>2018-05-29 22:45:02,705-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:45:02,718-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [5e8858aa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:45:02,719-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5e8858aa] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5aa864b3
>2018-05-29 22:45:02,722-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5e8858aa] FINISH, SetVdsStatusVDSCommand, log id: 5aa864b3
>2018-05-29 22:45:02,732-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e8858aa] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5e8858aa, Job ID: a5ef3ab4-fff7-4564-95bf-95de63558844, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:45:02,737-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e8858aa] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 442cb1f1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:45:02,739-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e8858aa] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 442cb1f1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:45:02,749-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [784505dc] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:45:02,751-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [784505dc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [648d8011] Autorecovering 2 hosts
>2018-05-29 22:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [648d8011] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:50:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b874010] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:50:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b874010] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:50:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b874010] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:50:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b874010] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:50:00,051-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7b874010] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f2f2a0a
>2018-05-29 22:50:00,053-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7b874010] FINISH, SetVdsStatusVDSCommand, log id: 1f2f2a0a
>2018-05-29 22:50:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b874010] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:50:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b874010] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:50:00,056-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7b874010] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:50:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3f37e476] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3f37e476] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3f37e476] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3f37e476] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:50:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3f37e476] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50189949
>2018-05-29 22:50:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3f37e476] FINISH, SetVdsStatusVDSCommand, log id: 50189949
>2018-05-29 22:50:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3f37e476] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 54ad8b1d
>2018-05-29 22:50:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3f37e476] FINISH, SetHaMaintenanceModeVDSCommand, log id: 54ad8b1d
>2018-05-29 22:50:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3f37e476] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:50:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3f37e476] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:50:00,351-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [34791797] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3e778e1e
>2018-05-29 22:50:00,353-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [34791797] FINISH, GetHardwareInfoVDSCommand, log id: 3e778e1e
>2018-05-29 22:50:00,366-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [108b8775] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:50:00,374-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [4d1a64ce] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:50:00,380-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [56dd39ef] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:50:00,386-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [56dd39ef] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: f8d97e4
>2018-05-29 22:50:00,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [56dd39ef] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:50:00,767-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [56dd39ef] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: f8d97e4
>2018-05-29 22:50:00,771-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [56dd39ef] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:50:00,772-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [56dd39ef] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:50:00,776-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [56dd39ef] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:50:00,777-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [56dd39ef] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:50:00,781-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [56dd39ef] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:50:00,781-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [56dd39ef] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:50:00,784-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [56dd39ef] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:50:00,787-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [56dd39ef] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 56dd39ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:50:00,790-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 17771ac4
>2018-05-29 22:50:00,790-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:50:00,887-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:50:00,887-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:50:00,887-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:50:00,887-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:50:00,887-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 17771ac4
>2018-05-29 22:50:00,888-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:50:00,901-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [65a809fa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:50:00,902-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [65a809fa] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46c3a4d9
>2018-05-29 22:50:00,904-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [65a809fa] FINISH, SetVdsStatusVDSCommand, log id: 46c3a4d9
>2018-05-29 22:50:00,934-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [65a809fa] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 65a809fa, Job ID: 5c911332-987e-4a35-9013-43e1cc8cd501, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:50:00,938-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [65a809fa] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4d1a64ce, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:50:00,940-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [65a809fa] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4d1a64ce, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:50:00,951-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [578d9163] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:50:00,952-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [578d9163] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:50:02,041-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [15ea1d36] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4062128f
>2018-05-29 22:50:02,044-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [15ea1d36] FINISH, GetHardwareInfoVDSCommand, log id: 4062128f
>2018-05-29 22:50:02,057-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [55b6459a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:50:02,066-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5ebae99a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:50:02,071-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [9ea8561] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:50:02,083-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [9ea8561] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 73cfb13
>2018-05-29 22:50:02,453-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [9ea8561] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:50:02,453-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [9ea8561] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 73cfb13
>2018-05-29 22:50:02,458-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [9ea8561] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:50:02,459-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [9ea8561] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:50:02,462-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [9ea8561] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:50:02,463-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [9ea8561] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:50:02,467-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [9ea8561] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:50:02,468-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [9ea8561] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:50:02,470-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [9ea8561] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:50:02,473-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [9ea8561] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 9ea8561, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:50:02,480-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 10f2d118
>2018-05-29 22:50:02,480-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:50:02,587-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:50:02,587-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:50:02,587-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:50:02,587-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:50:02,587-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 10f2d118
>2018-05-29 22:50:02,588-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:50:02,601-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [43e4f544] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:50:02,603-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [43e4f544] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1def0f61
>2018-05-29 22:50:02,605-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [43e4f544] FINISH, SetVdsStatusVDSCommand, log id: 1def0f61
>2018-05-29 22:50:02,637-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43e4f544] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 43e4f544, Job ID: 32ad51f0-b546-43c1-9538-dc0f758c8ced, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:50:02,641-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43e4f544] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5ebae99a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:50:02,644-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43e4f544] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5ebae99a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:50:02,655-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [599d1bca] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:50:02,656-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [599d1bca] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:54:43,332-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler5) [6130f79f] Attempting to update VMs/Templates Ovf.
>2018-05-29 22:54:43,335-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 22:54:43,335-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 22:54:43,335-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:54:43,383-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 22:54:43,387-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 22:54:43,387-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 22:54:43,387-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 22:54:43,387-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 22:54:43,388-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 22:54:43,389-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 22:55:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [19e28d44] Autorecovering 2 hosts
>2018-05-29 22:55:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [19e28d44] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:55:00,033-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2df7abfc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:55:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2df7abfc] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:55:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2df7abfc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:55:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2df7abfc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:55:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2df7abfc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 679a8bb6
>2018-05-29 22:55:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2df7abfc] FINISH, SetVdsStatusVDSCommand, log id: 679a8bb6
>2018-05-29 22:55:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2df7abfc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:55:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2df7abfc] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 22:55:00,047-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2df7abfc] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:55:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6f4ecc9a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:55:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6f4ecc9a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 22:55:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6f4ecc9a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:55:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6f4ecc9a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:55:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6f4ecc9a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1dc25fc
>2018-05-29 22:55:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6f4ecc9a] FINISH, SetVdsStatusVDSCommand, log id: 1dc25fc
>2018-05-29 22:55:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [6f4ecc9a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3d7f389e
>2018-05-29 22:55:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [6f4ecc9a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3d7f389e
>2018-05-29 22:55:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6f4ecc9a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 22:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6f4ecc9a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 22:55:00,291-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [66361ebc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 72a22764
>2018-05-29 22:55:00,292-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [66361ebc] FINISH, GetHardwareInfoVDSCommand, log id: 72a22764
>2018-05-29 22:55:00,305-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4f391ac9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:55:00,309-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [642bd1d9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:55:00,315-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [c5e5262] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:55:00,327-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [c5e5262] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 11563dc4
>2018-05-29 22:55:00,706-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c5e5262] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:55:00,706-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [c5e5262] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 11563dc4
>2018-05-29 22:55:00,710-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c5e5262] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:55:00,710-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [c5e5262] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:55:00,714-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c5e5262] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:55:00,715-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [c5e5262] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:55:00,719-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c5e5262] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:55:00,720-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [c5e5262] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:55:00,722-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [c5e5262] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:55:00,725-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c5e5262] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: c5e5262, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:55:00,731-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2c6dde80
>2018-05-29 22:55:00,731-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:55:00,830-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:55:00,830-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:55:00,830-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:55:00,830-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:55:00,830-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2c6dde80
>2018-05-29 22:55:00,830-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:55:00,844-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [6b5fed9f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:55:00,845-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6b5fed9f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ec09f1f
>2018-05-29 22:55:00,847-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6b5fed9f] FINISH, SetVdsStatusVDSCommand, log id: 3ec09f1f
>2018-05-29 22:55:00,856-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b5fed9f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6b5fed9f, Job ID: 15f9eba5-9c5c-4bf7-8e0b-22b2dfed2665, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:55:00,861-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b5fed9f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 642bd1d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:55:00,863-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b5fed9f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 642bd1d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:55:00,874-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [110997c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 22:55:00,875-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [110997c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 22:55:01,534-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2f661022] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3cd6a371
>2018-05-29 22:55:01,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2f661022] FINISH, GetHardwareInfoVDSCommand, log id: 3cd6a371
>2018-05-29 22:55:01,549-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [48090f3a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:55:01,554-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [8d64f96] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:55:01,559-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4d4ba2a3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 22:55:01,571-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4d4ba2a3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: d65845f
>2018-05-29 22:55:01,949-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4d4ba2a3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 22:55:01,949-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4d4ba2a3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: d65845f
>2018-05-29 22:55:01,953-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4d4ba2a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:55:01,954-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4d4ba2a3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:55:01,958-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4d4ba2a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:55:01,958-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4d4ba2a3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:55:01,962-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4d4ba2a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 22:55:01,963-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4d4ba2a3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 22:55:01,966-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4d4ba2a3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 22:55:01,969-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4d4ba2a3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d4ba2a3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 22:55:01,976-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1b4334dc
>2018-05-29 22:55:01,977-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 22:55:02,073-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:55:02,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 22:55:02,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 22:55:02,073-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 22:55:02,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1b4334dc
>2018-05-29 22:55:02,073-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 22:55:02,086-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [69bf14b7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:55:02,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [69bf14b7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5db48c28
>2018-05-29 22:55:02,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [69bf14b7] FINISH, SetVdsStatusVDSCommand, log id: 5db48c28
>2018-05-29 22:55:02,098-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69bf14b7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 69bf14b7, Job ID: bcf7d22f-4c3f-426a-ae55-5f8edf4c6c47, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 22:55:02,103-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69bf14b7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 8d64f96, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 22:55:02,105-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69bf14b7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 8d64f96, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 22:55:02,115-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [31471ddf] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 22:55:02,116-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [31471ddf] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2fe644ce] Autorecovering 2 hosts
>2018-05-29 23:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2fe644ce] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:00:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3b6a02d1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3b6a02d1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3b6a02d1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:00:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3b6a02d1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:00:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3b6a02d1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 171edfdf
>2018-05-29 23:00:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3b6a02d1] FINISH, SetVdsStatusVDSCommand, log id: 171edfdf
>2018-05-29 23:00:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3b6a02d1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:00:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3b6a02d1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:00:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3b6a02d1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:00:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7c5037b8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7c5037b8] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7c5037b8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:00:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7c5037b8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:00:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7c5037b8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ca07405
>2018-05-29 23:00:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7c5037b8] FINISH, SetVdsStatusVDSCommand, log id: 6ca07405
>2018-05-29 23:00:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7c5037b8] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3685d3ba
>2018-05-29 23:00:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7c5037b8] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3685d3ba
>2018-05-29 23:00:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7c5037b8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:00:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7c5037b8] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:00:01,528-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1739f83d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5c918e95
>2018-05-29 23:00:01,530-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1739f83d] FINISH, GetHardwareInfoVDSCommand, log id: 5c918e95
>2018-05-29 23:00:01,543-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [5085d019] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:00:01,552-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1ced3c5d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:00:01,557-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [454263b6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:00:01,564-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [454263b6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 17fc72e
>2018-05-29 23:00:01,932-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [454263b6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:00:01,932-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [454263b6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 17fc72e
>2018-05-29 23:00:01,936-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [454263b6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:00:01,936-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [454263b6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:00:01,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [454263b6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:00:01,941-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [454263b6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:00:01,945-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [454263b6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:00:01,946-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [454263b6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:00:01,949-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [454263b6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:00:01,951-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [454263b6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 454263b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:00:01,955-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 28f4328
>2018-05-29 23:00:01,955-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:00:02,053-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:00:02,053-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:00:02,053-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:00:02,053-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:00:02,053-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 28f4328
>2018-05-29 23:00:02,053-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:00:02,067-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [53299d63] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:00:02,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [53299d63] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e1564b0
>2018-05-29 23:00:02,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [53299d63] FINISH, SetVdsStatusVDSCommand, log id: 5e1564b0
>2018-05-29 23:00:02,100-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [53299d63] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 53299d63, Job ID: b5858af7-2e42-4b32-8e04-d3f693e3eff9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:00:02,105-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [53299d63] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1ced3c5d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:00:02,107-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [53299d63] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1ced3c5d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:00:02,118-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [da44710] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:00:02,119-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [da44710] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:00:02,785-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3c47fac9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 284f6cc9
>2018-05-29 23:00:02,787-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3c47fac9] FINISH, GetHardwareInfoVDSCommand, log id: 284f6cc9
>2018-05-29 23:00:02,800-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [272a546c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:00:02,805-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [706953a6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:00:02,810-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [20907629] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:00:02,816-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [20907629] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 71af19fb
>2018-05-29 23:00:03,189-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20907629] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:00:03,189-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [20907629] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 71af19fb
>2018-05-29 23:00:03,193-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20907629] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:00:03,194-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [20907629] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:00:03,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20907629] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:00:03,199-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [20907629] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:00:03,203-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20907629] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:00:03,203-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [20907629] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:00:03,206-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [20907629] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:00:03,209-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20907629] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 20907629, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:00:03,211-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 50dafc7
>2018-05-29 23:00:03,211-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:00:03,306-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:00:03,306-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:00:03,306-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:00:03,306-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:00:03,306-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 50dafc7
>2018-05-29 23:00:03,306-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:00:03,318-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [34a68da3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:00:03,320-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [34a68da3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1c813ad2
>2018-05-29 23:00:03,322-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [34a68da3] FINISH, SetVdsStatusVDSCommand, log id: 1c813ad2
>2018-05-29 23:00:03,327-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [34a68da3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 34a68da3, Job ID: 28c96c64-92cc-4f0e-b482-ca5c04b84f61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:00:03,332-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [34a68da3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 706953a6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:00:03,334-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [34a68da3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 706953a6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:00:03,345-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [17b5e7ff] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:00:03,346-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [17b5e7ff] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [429a7988] Autorecovering 2 hosts
>2018-05-29 23:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [429a7988] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:05:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [158bcdee] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:05:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [158bcdee] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:05:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [158bcdee] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:05:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [158bcdee] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:05:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [158bcdee] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 561f9476
>2018-05-29 23:05:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [158bcdee] FINISH, SetVdsStatusVDSCommand, log id: 561f9476
>2018-05-29 23:05:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [158bcdee] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:05:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [158bcdee] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:05:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [158bcdee] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:05:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [68b71222] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:05:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [68b71222] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:05:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [68b71222] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:05:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [68b71222] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:05:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [68b71222] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39a68c53
>2018-05-29 23:05:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [68b71222] FINISH, SetVdsStatusVDSCommand, log id: 39a68c53
>2018-05-29 23:05:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [68b71222] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 61460479
>2018-05-29 23:05:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [68b71222] FINISH, SetHaMaintenanceModeVDSCommand, log id: 61460479
>2018-05-29 23:05:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [68b71222] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:05:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [68b71222] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:05:00,985-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [47f0c210] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6f8b4139
>2018-05-29 23:05:00,987-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [47f0c210] FINISH, GetHardwareInfoVDSCommand, log id: 6f8b4139
>2018-05-29 23:05:01,000-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [9364fb0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:05:01,009-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5a29b6f3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:05:01,014-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3521198f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:05:01,026-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3521198f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 32ac8e36
>2018-05-29 23:05:01,413-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3521198f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:05:01,413-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3521198f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 32ac8e36
>2018-05-29 23:05:01,417-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3521198f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:05:01,418-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3521198f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:05:01,422-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3521198f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:05:01,422-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3521198f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:05:01,426-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3521198f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:05:01,427-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3521198f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:05:01,429-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3521198f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:05:01,432-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3521198f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3521198f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:05:01,439-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6a602f25
>2018-05-29 23:05:01,439-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:05:01,540-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:05:01,540-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:05:01,540-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:05:01,540-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:05:01,540-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6a602f25
>2018-05-29 23:05:01,540-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:05:01,553-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [257119fb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:05:01,555-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [257119fb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25785063
>2018-05-29 23:05:01,557-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [257119fb] FINISH, SetVdsStatusVDSCommand, log id: 25785063
>2018-05-29 23:05:01,565-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [257119fb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 257119fb, Job ID: 37619be1-b9ba-45e1-83c1-1e197dea5420, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:05:01,570-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [257119fb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5a29b6f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:05:01,572-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [257119fb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5a29b6f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:05:01,583-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [60db34ea] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:05:01,585-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [60db34ea] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:05:02,820-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [60b0b864] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3789074b
>2018-05-29 23:05:02,822-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [60b0b864] FINISH, GetHardwareInfoVDSCommand, log id: 3789074b
>2018-05-29 23:05:02,836-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [fe77d2d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:05:02,840-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [4e444dbe] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:05:02,845-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [30c22210] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:05:02,852-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [30c22210] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6044ae78
>2018-05-29 23:05:03,227-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [30c22210] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:05:03,227-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [30c22210] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6044ae78
>2018-05-29 23:05:03,231-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [30c22210] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:05:03,232-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [30c22210] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:05:03,236-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [30c22210] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:05:03,237-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [30c22210] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:05:03,240-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [30c22210] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:05:03,241-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [30c22210] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:05:03,244-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [30c22210] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:05:03,246-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [30c22210] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 30c22210, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:05:03,249-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6594eccd
>2018-05-29 23:05:03,249-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:05:03,348-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:05:03,348-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:05:03,348-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:05:03,348-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:05:03,348-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6594eccd
>2018-05-29 23:05:03,348-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:05:03,361-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [5888f42c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:05:03,362-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5888f42c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30f2fc89
>2018-05-29 23:05:03,364-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5888f42c] FINISH, SetVdsStatusVDSCommand, log id: 30f2fc89
>2018-05-29 23:05:03,370-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5888f42c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5888f42c, Job ID: fa6675aa-43dd-4daa-869b-ad76920f2900, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:05:03,374-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5888f42c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4e444dbe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:05:03,376-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5888f42c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4e444dbe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:05:03,387-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [6b3876bc] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:05:03,388-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [6b3876bc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [4cfd9818] Autorecovering 2 hosts
>2018-05-29 23:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [4cfd9818] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:10:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [216a2c39] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:10:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [216a2c39] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:10:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [216a2c39] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:10:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [216a2c39] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:10:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [216a2c39] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4203ff6d
>2018-05-29 23:10:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [216a2c39] FINISH, SetVdsStatusVDSCommand, log id: 4203ff6d
>2018-05-29 23:10:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [216a2c39] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:10:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [216a2c39] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:10:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [216a2c39] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:10:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [79d51413] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:10:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [79d51413] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:10:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [79d51413] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:10:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [79d51413] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:10:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [79d51413] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4caa3e95
>2018-05-29 23:10:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [79d51413] FINISH, SetVdsStatusVDSCommand, log id: 4caa3e95
>2018-05-29 23:10:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [79d51413] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 70eaaadb
>2018-05-29 23:10:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [79d51413] FINISH, SetHaMaintenanceModeVDSCommand, log id: 70eaaadb
>2018-05-29 23:10:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [79d51413] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:10:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [79d51413] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:10:01,029-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [14132cc7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 161f261f
>2018-05-29 23:10:01,031-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [14132cc7] FINISH, GetHardwareInfoVDSCommand, log id: 161f261f
>2018-05-29 23:10:01,043-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [1f843e3e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:10:01,051-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [4956cff1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:10:01,057-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4906a7e5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:10:01,068-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4906a7e5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2608a626
>2018-05-29 23:10:01,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4906a7e5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:10:01,436-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4906a7e5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2608a626
>2018-05-29 23:10:01,440-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4906a7e5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:10:01,441-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4906a7e5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:10:01,445-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4906a7e5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:10:01,446-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4906a7e5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:10:01,450-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4906a7e5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:10:01,451-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4906a7e5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:10:01,453-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4906a7e5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:10:01,456-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4906a7e5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4906a7e5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:10:01,462-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: a8897e7
>2018-05-29 23:10:01,462-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:10:01,559-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:10:01,559-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:10:01,559-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:10:01,559-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:10:01,559-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: a8897e7
>2018-05-29 23:10:01,559-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:10:01,572-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [439ec386] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:10:01,573-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [439ec386] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31ced8bd
>2018-05-29 23:10:01,576-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [439ec386] FINISH, SetVdsStatusVDSCommand, log id: 31ced8bd
>2018-05-29 23:10:01,607-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [439ec386] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 439ec386, Job ID: 44467d70-2d09-4b0f-9f49-33c63556d7d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:10:01,612-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [439ec386] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4956cff1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:10:01,614-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [439ec386] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4956cff1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:10:01,625-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [23b83695] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:10:01,626-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [23b83695] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:10:02,222-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [115edcaa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 797de36
>2018-05-29 23:10:02,223-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [115edcaa] FINISH, GetHardwareInfoVDSCommand, log id: 797de36
>2018-05-29 23:10:02,238-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [d91b2eb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:10:02,242-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [55bc84] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:10:02,248-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [403105e9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:10:02,255-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [403105e9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 38daba93
>2018-05-29 23:10:02,630-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [403105e9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:10:02,630-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [403105e9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 38daba93
>2018-05-29 23:10:02,634-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [403105e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:10:02,635-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [403105e9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:10:02,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [403105e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:10:02,639-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [403105e9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:10:02,643-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [403105e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:10:02,644-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [403105e9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:10:02,647-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [403105e9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:10:02,649-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [403105e9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 403105e9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:10:02,652-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6c7dec0c
>2018-05-29 23:10:02,652-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:10:02,746-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:10:02,746-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:10:02,746-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:10:02,746-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:10:02,746-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6c7dec0c
>2018-05-29 23:10:02,746-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:10:02,760-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [7aaa301f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:10:02,761-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7aaa301f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 306aa827
>2018-05-29 23:10:02,763-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7aaa301f] FINISH, SetVdsStatusVDSCommand, log id: 306aa827
>2018-05-29 23:10:02,769-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7aaa301f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7aaa301f, Job ID: 90444b6f-1bdf-4399-8fd1-34c857c0d9fc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:10:02,773-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7aaa301f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 55bc84, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:10:02,775-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7aaa301f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 55bc84, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:10:02,785-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [7cc15db] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:10:02,787-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [7cc15db] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [34c00909] Autorecovering 2 hosts
>2018-05-29 23:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [34c00909] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:15:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146e8e65] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:15:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146e8e65] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:15:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146e8e65] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:15:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146e8e65] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:15:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [146e8e65] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d28ee2c
>2018-05-29 23:15:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [146e8e65] FINISH, SetVdsStatusVDSCommand, log id: 3d28ee2c
>2018-05-29 23:15:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146e8e65] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:15:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146e8e65] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:15:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [146e8e65] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:15:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30e9e6e2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30e9e6e2] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30e9e6e2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30e9e6e2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:15:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [30e9e6e2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2675e980
>2018-05-29 23:15:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [30e9e6e2] FINISH, SetVdsStatusVDSCommand, log id: 2675e980
>2018-05-29 23:15:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [30e9e6e2] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6ceb8d27
>2018-05-29 23:15:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [30e9e6e2] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6ceb8d27
>2018-05-29 23:15:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30e9e6e2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:15:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30e9e6e2] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:15:00,402-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5b882be0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5a783d89
>2018-05-29 23:15:00,403-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5b882be0] FINISH, GetHardwareInfoVDSCommand, log id: 5a783d89
>2018-05-29 23:15:00,417-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [199d431b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:15:00,425-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [3103910] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:15:00,430-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [7addcfc9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:15:00,441-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [7addcfc9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 315c8a60
>2018-05-29 23:15:00,816-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7addcfc9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:15:00,816-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [7addcfc9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 315c8a60
>2018-05-29 23:15:00,820-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7addcfc9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:15:00,821-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [7addcfc9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:15:00,825-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7addcfc9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:15:00,826-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [7addcfc9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:15:00,830-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7addcfc9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:15:00,830-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [7addcfc9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:15:00,833-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [7addcfc9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:15:00,836-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7addcfc9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7addcfc9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:15:00,843-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1bce395c
>2018-05-29 23:15:00,843-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:15:00,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:15:00,940-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:15:00,940-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:15:00,940-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:15:00,940-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1bce395c
>2018-05-29 23:15:00,940-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:15:00,954-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [264e36ab] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:15:00,955-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [264e36ab] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7a81d665
>2018-05-29 23:15:00,957-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [264e36ab] FINISH, SetVdsStatusVDSCommand, log id: 7a81d665
>2018-05-29 23:15:00,989-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [264e36ab] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 264e36ab, Job ID: 5b53b42f-4b28-481d-a508-37b7ddf07964, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:15:00,994-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [264e36ab] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3103910, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:15:00,996-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [264e36ab] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3103910, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:15:01,007-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [1854937b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:15:01,008-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [1854937b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:15:02,331-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7a3dee80] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 564767a8
>2018-05-29 23:15:02,332-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7a3dee80] FINISH, GetHardwareInfoVDSCommand, log id: 564767a8
>2018-05-29 23:15:02,345-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3cffc9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:15:02,353-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [6830a04] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:15:02,358-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [336a28f9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:15:02,365-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [336a28f9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6b20c4e6
>2018-05-29 23:15:02,736-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [336a28f9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:15:02,736-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [336a28f9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6b20c4e6
>2018-05-29 23:15:02,740-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [336a28f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:15:02,741-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [336a28f9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:15:02,744-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [336a28f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:15:02,745-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [336a28f9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:15:02,749-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [336a28f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:15:02,749-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [336a28f9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:15:02,752-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [336a28f9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:15:02,754-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [336a28f9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 336a28f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:15:02,757-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 19c49875
>2018-05-29 23:15:02,757-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:15:02,853-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:15:02,853-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:15:02,853-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:15:02,853-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:15:02,853-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 19c49875
>2018-05-29 23:15:02,853-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:15:02,865-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [38c084d0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:15:02,866-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [38c084d0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6577f2ce
>2018-05-29 23:15:02,868-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [38c084d0] FINISH, SetVdsStatusVDSCommand, log id: 6577f2ce
>2018-05-29 23:15:02,898-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38c084d0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 38c084d0, Job ID: 867b872a-a246-49b1-8835-09b2a76ef64f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:15:02,902-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38c084d0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6830a04, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:15:02,904-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38c084d0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6830a04, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:15:02,915-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [4dc9ea16] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:15:02,916-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4dc9ea16] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:20:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [68ce616c] Autorecovering 2 hosts
>2018-05-29 23:20:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [68ce616c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:20:00,032-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [56c9abcb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:20:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [56c9abcb] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:20:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [56c9abcb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:20:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [56c9abcb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:20:00,041-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [56c9abcb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 63cf119b
>2018-05-29 23:20:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [56c9abcb] FINISH, SetVdsStatusVDSCommand, log id: 63cf119b
>2018-05-29 23:20:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [56c9abcb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:20:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [56c9abcb] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:20:00,046-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [56c9abcb] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:20:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [176238ca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:20:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [176238ca] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:20:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [176238ca] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:20:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [176238ca] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:20:00,060-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [176238ca] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e33a99
>2018-05-29 23:20:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [176238ca] FINISH, SetVdsStatusVDSCommand, log id: 5e33a99
>2018-05-29 23:20:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [176238ca] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 403fdb1b
>2018-05-29 23:20:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [176238ca] FINISH, SetHaMaintenanceModeVDSCommand, log id: 403fdb1b
>2018-05-29 23:20:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [176238ca] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [176238ca] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:20:00,621-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [a558a99] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1151436a
>2018-05-29 23:20:00,623-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [a558a99] FINISH, GetHardwareInfoVDSCommand, log id: 1151436a
>2018-05-29 23:20:00,635-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [797d1e0f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:20:00,640-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [c56deeb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:20:00,645-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [404f6c45] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:20:00,705-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [404f6c45] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 69081cb7
>2018-05-29 23:20:01,620-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [67ffaf4a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 29337058
>2018-05-29 23:20:01,622-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [67ffaf4a] FINISH, GetHardwareInfoVDSCommand, log id: 29337058
>2018-05-29 23:20:01,635-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [38083949] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:20:01,639-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [4cfce002] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:20:01,645-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [39e53779] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:20:01,651-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [39e53779] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 39abbec4
>2018-05-29 23:20:02,035-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [39e53779] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:20:02,035-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [39e53779] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 39abbec4
>2018-05-29 23:20:02,038-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [39e53779] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:20:02,039-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [39e53779] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:20:02,043-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [39e53779] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:20:02,044-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [39e53779] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:20:02,048-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [39e53779] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:20:02,048-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [39e53779] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:20:02,051-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [39e53779] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:20:02,054-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [39e53779] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 39e53779, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:20:02,061-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1565fd64
>2018-05-29 23:20:02,061-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:20:02,071-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [404f6c45] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:20:02,071-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [404f6c45] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 69081cb7
>2018-05-29 23:20:02,075-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [404f6c45] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:20:02,076-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [404f6c45] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:20:02,080-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [404f6c45] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:20:02,081-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [404f6c45] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:20:02,085-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [404f6c45] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:20:02,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [404f6c45] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:20:02,088-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [404f6c45] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:20:02,091-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [404f6c45] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 404f6c45, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:20:02,160-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:20:02,160-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:20:02,160-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:20:02,160-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:20:02,160-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1565fd64
>2018-05-29 23:20:02,160-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:20:02,162-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 595dfbff
>2018-05-29 23:20:02,162-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:20:02,173-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [6676868f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:20:02,175-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6676868f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5764380
>2018-05-29 23:20:02,177-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6676868f] FINISH, SetVdsStatusVDSCommand, log id: 5764380
>2018-05-29 23:20:02,185-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6676868f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6676868f, Job ID: 8d28deed-dea3-4db7-9d4c-75503568353c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:20:02,190-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6676868f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4cfce002, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:20:02,192-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6676868f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4cfce002, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:20:02,202-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [634315a2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:20:02,203-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [634315a2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:20:02,260-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:20:02,260-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:20:02,260-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:20:02,260-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:20:02,260-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 595dfbff
>2018-05-29 23:20:02,260-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:20:02,272-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [3c713291] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:20:02,273-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3c713291] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13ff759d
>2018-05-29 23:20:02,276-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3c713291] FINISH, SetVdsStatusVDSCommand, log id: 13ff759d
>2018-05-29 23:20:02,285-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c713291] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c713291, Job ID: ca6f88e6-f420-4751-a4f8-84d1694b6a2d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:20:02,289-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c713291] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: c56deeb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:20:02,291-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c713291] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: c56deeb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:20:02,302-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [2b10faea] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:20:02,303-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [2b10faea] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2ebe75e6] Autorecovering 2 hosts
>2018-05-29 23:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2ebe75e6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:25:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6274e364] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6274e364] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6274e364] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6274e364] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:25:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6274e364] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1c865971
>2018-05-29 23:25:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6274e364] FINISH, SetVdsStatusVDSCommand, log id: 1c865971
>2018-05-29 23:25:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6274e364] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:25:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6274e364] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:25:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6274e364] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:25:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58db5caf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58db5caf] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58db5caf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58db5caf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:25:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [58db5caf] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 62d9f030
>2018-05-29 23:25:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [58db5caf] FINISH, SetVdsStatusVDSCommand, log id: 62d9f030
>2018-05-29 23:25:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [58db5caf] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6eddd838
>2018-05-29 23:25:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [58db5caf] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6eddd838
>2018-05-29 23:25:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58db5caf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:25:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58db5caf] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:25:02,777-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [2ed6f859] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 207f8e5f
>2018-05-29 23:25:02,779-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [2ed6f859] FINISH, GetHardwareInfoVDSCommand, log id: 207f8e5f
>2018-05-29 23:25:02,792-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [4539c11b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:25:02,800-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [745cc989] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:25:02,805-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [63ea04f6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:25:02,817-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [63ea04f6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1fd0ff41
>2018-05-29 23:25:02,986-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [63d69cf3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6f772e42
>2018-05-29 23:25:02,988-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [63d69cf3] FINISH, GetHardwareInfoVDSCommand, log id: 6f772e42
>2018-05-29 23:25:03,001-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [42f60e52] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:25:03,005-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1b06ae05] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:25:03,010-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2fecabff] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:25:03,016-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2fecabff] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5227cd32
>2018-05-29 23:25:03,192-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63ea04f6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:25:03,192-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [63ea04f6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1fd0ff41
>2018-05-29 23:25:03,196-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63ea04f6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:25:03,197-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [63ea04f6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:25:03,200-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63ea04f6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:25:03,201-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [63ea04f6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:25:03,205-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63ea04f6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:25:03,206-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [63ea04f6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:25:03,208-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [63ea04f6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:25:03,211-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63ea04f6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 63ea04f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:25:03,214-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 447a5282
>2018-05-29 23:25:03,214-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:25:03,308-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:25:03,308-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:25:03,308-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:25:03,308-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:25:03,308-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 447a5282
>2018-05-29 23:25:03,308-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:25:03,321-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [679e9bf2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:25:03,322-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [679e9bf2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bbf1ec2
>2018-05-29 23:25:03,324-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [679e9bf2] FINISH, SetVdsStatusVDSCommand, log id: 3bbf1ec2
>2018-05-29 23:25:03,354-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [679e9bf2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 679e9bf2, Job ID: 5c7aaaf8-6570-49ce-8cee-44c642264e0b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:25:03,358-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [679e9bf2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 745cc989, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:25:03,361-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [679e9bf2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 745cc989, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:25:03,371-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [3edd0556] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:25:03,372-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [3edd0556] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:25:03,380-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2fecabff] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:25:03,380-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2fecabff] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5227cd32
>2018-05-29 23:25:03,384-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2fecabff] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:25:03,385-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2fecabff] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:25:03,389-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2fecabff] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:25:03,390-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2fecabff] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:25:03,393-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2fecabff] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:25:03,394-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2fecabff] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:25:03,397-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2fecabff] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:25:03,400-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2fecabff] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2fecabff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:25:03,403-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 75b9211a
>2018-05-29 23:25:03,403-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:25:03,504-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:25:03,504-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:25:03,504-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:25:03,504-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:25:03,504-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 75b9211a
>2018-05-29 23:25:03,504-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:25:03,517-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [584deb2e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:25:03,519-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [584deb2e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 55a3d645
>2018-05-29 23:25:03,521-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [584deb2e] FINISH, SetVdsStatusVDSCommand, log id: 55a3d645
>2018-05-29 23:25:03,526-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [584deb2e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 584deb2e, Job ID: ee3388f2-e189-4120-ab46-0eeb3e77f9f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:25:03,531-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [584deb2e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1b06ae05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:25:03,533-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [584deb2e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1b06ae05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:25:03,544-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2d02ad05] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:25:03,545-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2d02ad05] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [45ec4323] Autorecovering 2 hosts
>2018-05-29 23:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [45ec4323] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:30:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7c21dd18] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:30:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7c21dd18] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:30:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7c21dd18] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:30:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7c21dd18] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:30:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7c21dd18] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 58987790
>2018-05-29 23:30:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7c21dd18] FINISH, SetVdsStatusVDSCommand, log id: 58987790
>2018-05-29 23:30:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7c21dd18] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:30:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7c21dd18] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:30:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7c21dd18] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:30:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50b719d9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:30:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50b719d9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:30:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50b719d9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:30:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50b719d9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:30:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [50b719d9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 556bddf5
>2018-05-29 23:30:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [50b719d9] FINISH, SetVdsStatusVDSCommand, log id: 556bddf5
>2018-05-29 23:30:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [50b719d9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7ced0f6
>2018-05-29 23:30:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [50b719d9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7ced0f6
>2018-05-29 23:30:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50b719d9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:30:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50b719d9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:30:00,964-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2acaea5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4998301
>2018-05-29 23:30:00,966-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2acaea5] FINISH, GetHardwareInfoVDSCommand, log id: 4998301
>2018-05-29 23:30:00,978-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [592e1a35] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:30:00,987-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [24207665] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:30:00,992-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [229caa9d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:30:00,998-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [229caa9d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 598b03ea
>2018-05-29 23:30:01,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [43e45caa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 68b25925
>2018-05-29 23:30:01,180-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [43e45caa] FINISH, GetHardwareInfoVDSCommand, log id: 68b25925
>2018-05-29 23:30:01,192-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [5ffc4f92] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:30:01,196-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [4a42e16b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:30:01,203-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2d9cbedd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:30:01,209-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2d9cbedd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6e329349
>2018-05-29 23:30:01,381-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [229caa9d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:30:01,381-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [229caa9d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 598b03ea
>2018-05-29 23:30:01,385-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [229caa9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:30:01,386-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [229caa9d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:30:01,389-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [229caa9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:30:01,390-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [229caa9d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:30:01,394-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [229caa9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:30:01,394-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [229caa9d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:30:01,397-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [229caa9d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:30:01,399-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [229caa9d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 229caa9d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:30:01,406-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5a3b7700
>2018-05-29 23:30:01,406-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:30:01,504-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:30:01,504-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:30:01,504-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:30:01,504-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:30:01,504-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5a3b7700
>2018-05-29 23:30:01,504-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:30:01,518-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [39b90157] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:30:01,519-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [39b90157] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4ca84b9c
>2018-05-29 23:30:01,521-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [39b90157] FINISH, SetVdsStatusVDSCommand, log id: 4ca84b9c
>2018-05-29 23:30:01,553-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [39b90157] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 39b90157, Job ID: 4dc2f37a-dde2-4c0c-bd59-732d62db6b32, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:30:01,558-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [39b90157] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 24207665, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:30:01,560-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [39b90157] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 24207665, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:30:01,571-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [455974ad] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:30:01,572-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [455974ad] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:30:02,580-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2d9cbedd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:30:02,580-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2d9cbedd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6e329349
>2018-05-29 23:30:02,584-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2d9cbedd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:30:02,585-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2d9cbedd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:30:02,589-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2d9cbedd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:30:02,590-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2d9cbedd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:30:02,593-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2d9cbedd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:30:02,594-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2d9cbedd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:30:02,598-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2d9cbedd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:30:02,600-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2d9cbedd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2d9cbedd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:30:02,603-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4445e592
>2018-05-29 23:30:02,603-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:30:02,699-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:30:02,699-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:30:02,699-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:30:02,699-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:30:02,699-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4445e592
>2018-05-29 23:30:02,699-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:30:02,712-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [7777937d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:30:02,714-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7777937d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7e1304af
>2018-05-29 23:30:02,715-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7777937d] FINISH, SetVdsStatusVDSCommand, log id: 7e1304af
>2018-05-29 23:30:02,721-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7777937d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7777937d, Job ID: a7eeb39b-1c88-41da-8a5d-82cb2cd833d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:30:02,725-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7777937d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4a42e16b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:30:02,727-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7777937d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4a42e16b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:30:02,738-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [47a50b34] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:30:02,739-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [47a50b34] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:35:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [10f712b1] Autorecovering 2 hosts
>2018-05-29 23:35:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [10f712b1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:35:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [86b99f9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:35:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [86b99f9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [86b99f9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [86b99f9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:35:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [86b99f9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3036d278
>2018-05-29 23:35:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [86b99f9] FINISH, SetVdsStatusVDSCommand, log id: 3036d278
>2018-05-29 23:35:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [86b99f9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:35:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [86b99f9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:35:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [86b99f9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:35:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [10161227] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:35:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [10161227] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:35:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [10161227] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:35:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [10161227] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:35:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [10161227] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 15630d75
>2018-05-29 23:35:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [10161227] FINISH, SetVdsStatusVDSCommand, log id: 15630d75
>2018-05-29 23:35:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [10161227] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1f69f59a
>2018-05-29 23:35:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [10161227] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1f69f59a
>2018-05-29 23:35:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [10161227] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:35:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [10161227] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:35:00,431-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [14763349] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4e843cb5
>2018-05-29 23:35:00,432-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [14763349] FINISH, GetHardwareInfoVDSCommand, log id: 4e843cb5
>2018-05-29 23:35:00,446-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [3282da15] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:35:00,454-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [217f4052] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:35:00,459-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [d039538] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:35:00,470-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [d039538] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5e12f30
>2018-05-29 23:35:00,844-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [d039538] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:35:00,844-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [d039538] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5e12f30
>2018-05-29 23:35:00,848-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [d039538] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:35:00,849-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [d039538] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:35:00,852-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [d039538] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:35:00,853-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [d039538] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:35:00,857-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [d039538] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:35:00,858-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [d039538] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:35:00,861-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [d039538] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:35:00,863-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [d039538] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: d039538, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:35:00,870-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2d98d9e8
>2018-05-29 23:35:00,870-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:35:00,967-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:35:00,967-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:35:00,967-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:35:00,967-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:35:00,967-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2d98d9e8
>2018-05-29 23:35:00,967-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:35:00,981-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [3d6b2487] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:35:00,982-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3d6b2487] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6b595b2c
>2018-05-29 23:35:00,984-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3d6b2487] FINISH, SetVdsStatusVDSCommand, log id: 6b595b2c
>2018-05-29 23:35:01,016-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d6b2487] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3d6b2487, Job ID: 7fbf84f7-87e5-4797-ac7d-51a1fd1bd0a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:35:01,020-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d6b2487] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 217f4052, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:35:01,023-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d6b2487] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 217f4052, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:35:01,034-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [59e33220] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:35:01,035-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [59e33220] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:35:02,202-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [4e3c806b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 36872b4c
>2018-05-29 23:35:02,204-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [4e3c806b] FINISH, GetHardwareInfoVDSCommand, log id: 36872b4c
>2018-05-29 23:35:02,217-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [5385c976] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:35:02,225-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [5e879d08] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:35:02,231-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [75834642] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:35:02,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [75834642] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 570a195f
>2018-05-29 23:35:02,671-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75834642] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:35:02,671-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [75834642] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 570a195f
>2018-05-29 23:35:02,675-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75834642] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:35:02,676-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [75834642] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:35:02,679-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75834642] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:35:02,680-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [75834642] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:35:02,684-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75834642] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:35:02,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [75834642] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:35:02,687-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [75834642] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:35:02,690-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75834642] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 75834642, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:35:02,696-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: af16986
>2018-05-29 23:35:02,696-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:35:02,793-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:35:02,793-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:35:02,793-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:35:02,793-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:35:02,793-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: af16986
>2018-05-29 23:35:02,793-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:35:02,806-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [2a7a0ff0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:35:02,807-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2a7a0ff0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2eb2109a
>2018-05-29 23:35:02,809-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2a7a0ff0] FINISH, SetVdsStatusVDSCommand, log id: 2eb2109a
>2018-05-29 23:35:02,843-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2a7a0ff0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2a7a0ff0, Job ID: 0c0674f9-6912-4753-9ccb-e26c695a8e63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:35:02,847-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2a7a0ff0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5e879d08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:35:02,849-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2a7a0ff0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5e879d08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:35:02,861-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [6f93bc0d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:35:02,862-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [6f93bc0d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:40:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [ea8ee91] Autorecovering 2 hosts
>2018-05-29 23:40:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [ea8ee91] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:40:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f3ef80b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f3ef80b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f3ef80b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f3ef80b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:40:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3f3ef80b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 65b31a49
>2018-05-29 23:40:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3f3ef80b] FINISH, SetVdsStatusVDSCommand, log id: 65b31a49
>2018-05-29 23:40:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f3ef80b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:40:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f3ef80b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:40:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3f3ef80b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:40:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4bab52e0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:40:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4bab52e0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:40:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4bab52e0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:40:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4bab52e0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:40:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4bab52e0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 70991d89
>2018-05-29 23:40:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4bab52e0] FINISH, SetVdsStatusVDSCommand, log id: 70991d89
>2018-05-29 23:40:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4bab52e0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3a2e56bb
>2018-05-29 23:40:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4bab52e0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3a2e56bb
>2018-05-29 23:40:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4bab52e0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:40:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4bab52e0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:40:00,448-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4c406f80] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5667e0e1
>2018-05-29 23:40:00,449-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4c406f80] FINISH, GetHardwareInfoVDSCommand, log id: 5667e0e1
>2018-05-29 23:40:00,462-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [7dc8b9ce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:40:00,470-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [4f2135e2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:40:00,476-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [786aaf00] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:40:00,482-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [786aaf00] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5be6a8be
>2018-05-29 23:40:00,862-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [786aaf00] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:40:00,862-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [786aaf00] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5be6a8be
>2018-05-29 23:40:00,866-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [786aaf00] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:40:00,867-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [786aaf00] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:40:00,871-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [786aaf00] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:40:00,872-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [786aaf00] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:40:00,876-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [786aaf00] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:40:00,876-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [786aaf00] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:40:00,879-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [786aaf00] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:40:00,882-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [786aaf00] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 786aaf00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:40:00,884-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7efbb47b
>2018-05-29 23:40:00,884-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:40:00,980-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:40:00,980-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:40:00,981-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:40:00,981-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:40:00,981-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7efbb47b
>2018-05-29 23:40:00,981-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:40:00,994-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [586793f2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:40:00,995-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [586793f2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19c2f532
>2018-05-29 23:40:00,997-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [586793f2] FINISH, SetVdsStatusVDSCommand, log id: 19c2f532
>2018-05-29 23:40:01,026-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [586793f2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 586793f2, Job ID: 21459b2a-56b5-40d2-9767-b0f912f45bab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:40:01,031-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [586793f2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4f2135e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:40:01,033-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [586793f2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4f2135e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:40:01,045-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [562dc36c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:40:01,046-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [562dc36c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:40:01,732-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [10ac7aae] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 385e4d34
>2018-05-29 23:40:01,734-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [10ac7aae] FINISH, GetHardwareInfoVDSCommand, log id: 385e4d34
>2018-05-29 23:40:01,746-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [5088f758] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:40:01,751-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [7af001db] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:40:01,756-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [306c195e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:40:01,762-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [306c195e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5b890f38
>2018-05-29 23:40:03,130-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [306c195e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:40:03,130-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [306c195e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5b890f38
>2018-05-29 23:40:03,134-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [306c195e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:40:03,135-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [306c195e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:40:03,139-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [306c195e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:40:03,140-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [306c195e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:40:03,144-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [306c195e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:40:03,144-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [306c195e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:40:03,148-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [306c195e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:40:03,150-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [306c195e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 306c195e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:40:03,153-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4883c357
>2018-05-29 23:40:03,153-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:40:03,248-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:40:03,248-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:40:03,248-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:40:03,248-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:40:03,248-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4883c357
>2018-05-29 23:40:03,248-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:40:03,262-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [354bf78e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:40:03,263-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [354bf78e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5efe377f
>2018-05-29 23:40:03,265-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [354bf78e] FINISH, SetVdsStatusVDSCommand, log id: 5efe377f
>2018-05-29 23:40:03,296-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [354bf78e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 354bf78e, Job ID: 59cbd78c-4924-4745-aea0-5f8ecda4bb1b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:40:03,300-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [354bf78e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7af001db, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:40:03,302-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [354bf78e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7af001db, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:40:03,313-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [6092ff92] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:40:03,314-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [6092ff92] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:45:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [29aa5cc0] Autorecovering 2 hosts
>2018-05-29 23:45:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [29aa5cc0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:45:00,032-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b323604] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:45:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b323604] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:45:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b323604] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:45:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b323604] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:45:00,041-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5b323604] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7dd08377
>2018-05-29 23:45:00,043-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5b323604] FINISH, SetVdsStatusVDSCommand, log id: 7dd08377
>2018-05-29 23:45:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b323604] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:45:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b323604] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:45:00,046-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5b323604] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:45:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [75f721a4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:45:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [75f721a4] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:45:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [75f721a4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:45:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [75f721a4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:45:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [75f721a4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2bbd194d
>2018-05-29 23:45:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [75f721a4] FINISH, SetVdsStatusVDSCommand, log id: 2bbd194d
>2018-05-29 23:45:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [75f721a4] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5a19488b
>2018-05-29 23:45:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [75f721a4] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5a19488b
>2018-05-29 23:45:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [75f721a4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:45:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [75f721a4] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:45:00,979-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6d2d68f3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7070036a
>2018-05-29 23:45:00,981-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6d2d68f3] FINISH, GetHardwareInfoVDSCommand, log id: 7070036a
>2018-05-29 23:45:00,994-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [47de66ae] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:45:00,999-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [37d56549] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:45:01,004-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [158553bf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:45:01,015-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [158553bf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 50e4992e
>2018-05-29 23:45:01,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [158553bf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:45:01,388-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [158553bf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 50e4992e
>2018-05-29 23:45:01,392-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [158553bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:45:01,393-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [158553bf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:45:01,397-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [158553bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:45:01,397-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [158553bf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:45:01,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [158553bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:45:01,402-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [158553bf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:45:01,404-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [158553bf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:45:01,407-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [158553bf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 158553bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:45:01,414-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 680a266
>2018-05-29 23:45:01,414-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:45:01,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-11) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:45:01,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:45:01,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:45:01,513-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:45:01,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] FINISH, ConnectStoragePoolVDSCommand, log id: 680a266
>2018-05-29 23:45:01,513-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-11) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:45:01,526-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [4c725d0e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:45:01,527-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4c725d0e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e61c1d6
>2018-05-29 23:45:01,530-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4c725d0e] FINISH, SetVdsStatusVDSCommand, log id: 5e61c1d6
>2018-05-29 23:45:01,538-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c725d0e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4c725d0e, Job ID: 6f54f9db-da39-4d7c-af5a-41df99b57527, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:45:01,543-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c725d0e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 37d56549, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:45:01,545-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c725d0e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 37d56549, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:45:01,556-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [4d1a3ff2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:45:01,557-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [4d1a3ff2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:45:01,596-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [1754d4ea] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 60c6afb
>2018-05-29 23:45:01,598-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [1754d4ea] FINISH, GetHardwareInfoVDSCommand, log id: 60c6afb
>2018-05-29 23:45:01,611-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [1939e03] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:45:01,615-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [6722f966] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:45:01,620-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2975b27a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:45:01,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2975b27a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 73ae7238
>2018-05-29 23:45:02,010-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2975b27a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:45:02,010-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2975b27a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 73ae7238
>2018-05-29 23:45:02,014-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2975b27a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:45:02,014-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2975b27a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:45:02,018-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2975b27a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:45:02,019-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2975b27a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:45:02,023-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2975b27a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:45:02,024-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2975b27a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:45:02,027-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2975b27a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:45:02,030-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2975b27a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2975b27a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:45:02,036-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 37973309
>2018-05-29 23:45:02,036-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:45:02,133-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:45:02,133-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:45:02,133-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:45:02,133-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:45:02,133-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 37973309
>2018-05-29 23:45:02,133-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:45:02,147-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [1121a086] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:45:02,148-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1121a086] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3c184491
>2018-05-29 23:45:02,151-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1121a086] FINISH, SetVdsStatusVDSCommand, log id: 3c184491
>2018-05-29 23:45:02,159-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1121a086] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1121a086, Job ID: fce44a74-3b66-42d9-a268-9c4a199f4226, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:45:02,164-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1121a086] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6722f966, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:45:02,166-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1121a086] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6722f966, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:45:02,177-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [7269cb79] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:45:02,178-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [7269cb79] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [5ff7375] Autorecovering 2 hosts
>2018-05-29 23:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [5ff7375] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:50:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [45cd4015] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:50:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [45cd4015] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:50:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [45cd4015] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:50:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [45cd4015] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:50:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [45cd4015] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5b51cff5
>2018-05-29 23:50:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [45cd4015] FINISH, SetVdsStatusVDSCommand, log id: 5b51cff5
>2018-05-29 23:50:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [45cd4015] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:50:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [45cd4015] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:50:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [45cd4015] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:50:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [4a74cb29] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:50:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [4a74cb29] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:50:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [4a74cb29] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:50:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [4a74cb29] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:50:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4a74cb29] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 510d57af
>2018-05-29 23:50:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4a74cb29] FINISH, SetVdsStatusVDSCommand, log id: 510d57af
>2018-05-29 23:50:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [4a74cb29] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 214a3ab1
>2018-05-29 23:50:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [4a74cb29] FINISH, SetHaMaintenanceModeVDSCommand, log id: 214a3ab1
>2018-05-29 23:50:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [4a74cb29] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:50:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [4a74cb29] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:50:02,379-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6778725e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1c43e048
>2018-05-29 23:50:02,381-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6778725e] FINISH, GetHardwareInfoVDSCommand, log id: 1c43e048
>2018-05-29 23:50:02,394-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [23b7507b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:50:02,402-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [4d72a1f0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:50:02,407-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2c7f3613] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:50:02,464-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2c7f3613] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7e053389
>2018-05-29 23:50:03,036-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [647894f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2abba7c7
>2018-05-29 23:50:03,038-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [647894f] FINISH, GetHardwareInfoVDSCommand, log id: 2abba7c7
>2018-05-29 23:50:03,051-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [22b675ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:50:03,056-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [72532b8a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:50:03,062-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6ea64f18] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:50:03,068-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6ea64f18] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 24b51893
>2018-05-29 23:50:03,452-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6ea64f18] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:50:03,452-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6ea64f18] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 24b51893
>2018-05-29 23:50:03,456-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6ea64f18] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:50:03,456-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6ea64f18] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:50:03,460-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6ea64f18] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:50:03,461-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6ea64f18] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:50:03,465-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6ea64f18] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:50:03,465-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6ea64f18] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:50:03,468-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6ea64f18] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:50:03,471-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6ea64f18] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6ea64f18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:50:03,478-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7e16e022
>2018-05-29 23:50:03,478-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:50:03,576-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:50:03,576-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:50:03,576-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:50:03,576-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:50:03,576-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7e16e022
>2018-05-29 23:50:03,576-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:50:03,589-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [7fd94ba9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:50:03,591-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7fd94ba9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d98d4a9
>2018-05-29 23:50:03,593-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7fd94ba9] FINISH, SetVdsStatusVDSCommand, log id: d98d4a9
>2018-05-29 23:50:03,625-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fd94ba9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7fd94ba9, Job ID: 37d6c997-5dcd-42e1-a509-9316bb10ea2f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:50:03,629-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fd94ba9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 72532b8a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:50:03,632-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fd94ba9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 72532b8a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:50:03,643-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [79d33512] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:50:03,644-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [79d33512] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:50:03,833-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2c7f3613] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:50:03,833-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2c7f3613] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7e053389
>2018-05-29 23:50:03,837-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2c7f3613] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:50:03,838-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2c7f3613] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:50:03,842-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2c7f3613] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:50:03,843-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2c7f3613] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:50:03,846-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2c7f3613] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:50:03,847-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2c7f3613] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:50:03,850-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2c7f3613] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:50:03,853-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2c7f3613] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2c7f3613, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:50:03,856-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1589d67a
>2018-05-29 23:50:03,856-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:50:03,952-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:50:03,952-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:50:03,952-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:50:03,952-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:50:03,953-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1589d67a
>2018-05-29 23:50:03,953-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:50:03,965-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [36ce228b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:50:03,966-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [36ce228b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5564f2d9
>2018-05-29 23:50:03,968-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [36ce228b] FINISH, SetVdsStatusVDSCommand, log id: 5564f2d9
>2018-05-29 23:50:03,974-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36ce228b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 36ce228b, Job ID: 3d768810-b9c4-4eb8-a281-5e148f56f2e6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:50:03,978-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36ce228b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4d72a1f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:50:03,980-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36ce228b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4d72a1f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:50:03,991-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [1e667b69] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:50:03,992-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1e667b69] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:54:43,389-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler7) [28c99cfe] Attempting to update VMs/Templates Ovf.
>2018-05-29 23:54:43,392-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 23:54:43,392-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 23:54:43,393-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:54:43,441-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-29 23:54:43,445-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-29 23:54:43,445-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Attempting to update template OVFs in Data Center 'Default'
>2018-05-29 23:54:43,445-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-29 23:54:43,445-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 23:54:43,446-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-29 23:54:43,447-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-29 23:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [13766868] Autorecovering 2 hosts
>2018-05-29 23:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [13766868] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:55:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [43af1542] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:55:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [43af1542] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:55:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [43af1542] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:55:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [43af1542] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:55:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [43af1542] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8cd26e3
>2018-05-29 23:55:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [43af1542] FINISH, SetVdsStatusVDSCommand, log id: 8cd26e3
>2018-05-29 23:55:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [43af1542] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:55:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [43af1542] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-29 23:55:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [43af1542] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:55:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7f25ba8d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:55:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7f25ba8d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-29 23:55:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7f25ba8d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:55:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7f25ba8d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:55:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7f25ba8d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6d2dcffc
>2018-05-29 23:55:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7f25ba8d] FINISH, SetVdsStatusVDSCommand, log id: 6d2dcffc
>2018-05-29 23:55:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [7f25ba8d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7b2e91fb
>2018-05-29 23:55:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [7f25ba8d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7b2e91fb
>2018-05-29 23:55:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7f25ba8d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-29 23:55:00,103-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7f25ba8d] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 7f25ba8d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-29 23:55:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7f25ba8d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-29 23:55:01,237-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [119348b2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5795d815
>2018-05-29 23:55:01,239-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [119348b2] FINISH, GetHardwareInfoVDSCommand, log id: 5795d815
>2018-05-29 23:55:01,254-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [72cad116] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:55:01,263-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [232e5ddf] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:55:01,268-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [759adb2b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:55:01,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [759adb2b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 525f7826
>2018-05-29 23:55:01,602-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2db31d46] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1fd116c1
>2018-05-29 23:55:01,603-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2db31d46] FINISH, GetHardwareInfoVDSCommand, log id: 1fd116c1
>2018-05-29 23:55:01,617-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [492600d0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:55:01,621-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [1cd643cd] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:55:01,626-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1c8c5c61] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-29 23:55:01,633-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1c8c5c61] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5b1d50e
>2018-05-29 23:55:01,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [759adb2b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:55:01,664-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [759adb2b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 525f7826
>2018-05-29 23:55:01,668-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [759adb2b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:55:01,669-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [759adb2b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:55:01,672-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [759adb2b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:55:01,673-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [759adb2b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:55:01,677-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [759adb2b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:55:01,678-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [759adb2b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:55:01,681-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [759adb2b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:55:01,683-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [759adb2b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 759adb2b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:55:01,690-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5fa99a2f
>2018-05-29 23:55:01,690-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:55:01,788-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:55:01,788-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:55:01,788-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:55:01,788-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:55:01,788-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5fa99a2f
>2018-05-29 23:55:01,788-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:55:01,802-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [791b48fd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:55:01,803-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [791b48fd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6afa846e
>2018-05-29 23:55:01,805-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [791b48fd] FINISH, SetVdsStatusVDSCommand, log id: 6afa846e
>2018-05-29 23:55:01,839-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [791b48fd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 791b48fd, Job ID: e373c20d-acd3-4ee3-bd75-b7074252caeb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:55:01,843-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [791b48fd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 232e5ddf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:55:01,845-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [791b48fd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 232e5ddf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:55:01,857-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [507c3c70] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-29 23:55:01,858-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [507c3c70] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-29 23:55:01,998-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1c8c5c61] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-29 23:55:01,998-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1c8c5c61] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5b1d50e
>2018-05-29 23:55:02,002-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1c8c5c61] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:55:02,003-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1c8c5c61] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:55:02,007-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1c8c5c61] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:55:02,008-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1c8c5c61] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:55:02,011-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1c8c5c61] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-29 23:55:02,012-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1c8c5c61] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-29 23:55:02,015-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1c8c5c61] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-29 23:55:02,017-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1c8c5c61] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1c8c5c61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-29 23:55:02,021-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 433788de
>2018-05-29 23:55:02,021-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-29 23:55:02,116-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:55:02,116-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-29 23:55:02,116-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-29 23:55:02,117-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-29 23:55:02,117-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 433788de
>2018-05-29 23:55:02,117-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-29 23:55:02,130-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [24ff9e9e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:55:02,131-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [24ff9e9e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 290e2136
>2018-05-29 23:55:02,134-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [24ff9e9e] FINISH, SetVdsStatusVDSCommand, log id: 290e2136
>2018-05-29 23:55:02,164-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24ff9e9e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 24ff9e9e, Job ID: f3aa4841-67e1-4078-93b2-5a3a59691911, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-29 23:55:02,168-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24ff9e9e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1cd643cd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-29 23:55:02,170-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24ff9e9e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1cd643cd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-29 23:55:02,181-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [2dc57252] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-29 23:55:02,182-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [2dc57252] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:00:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [49fb93e4] Autorecovering 2 hosts
>2018-05-30 00:00:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [49fb93e4] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:00:00,035-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5840b2f5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:00:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5840b2f5] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:00:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5840b2f5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:00:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5840b2f5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:00:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5840b2f5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f228e33
>2018-05-30 00:00:00,046-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5840b2f5] FINISH, SetVdsStatusVDSCommand, log id: 3f228e33
>2018-05-30 00:00:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5840b2f5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:00:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5840b2f5] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:00:00,049-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [5840b2f5] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:00:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21ab197c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:00:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21ab197c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:00:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21ab197c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:00:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21ab197c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:00:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [21ab197c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5118bb77
>2018-05-30 00:00:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [21ab197c] FINISH, SetVdsStatusVDSCommand, log id: 5118bb77
>2018-05-30 00:00:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [21ab197c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: af6eceb
>2018-05-30 00:00:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [21ab197c] FINISH, SetHaMaintenanceModeVDSCommand, log id: af6eceb
>2018-05-30 00:00:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21ab197c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:00:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21ab197c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:00:02,486-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [346dc253] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4b15c54a
>2018-05-30 00:00:02,488-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [346dc253] FINISH, GetHardwareInfoVDSCommand, log id: 4b15c54a
>2018-05-30 00:00:02,502-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4887976a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:00:02,506-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [45193dca] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:00:02,512-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7e3e8b41] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:00:02,524-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7e3e8b41] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7143fa8
>2018-05-30 00:00:02,817-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3c7f78be] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5a18a2bb
>2018-05-30 00:00:02,819-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3c7f78be] FINISH, GetHardwareInfoVDSCommand, log id: 5a18a2bb
>2018-05-30 00:00:02,832-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [90811b6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:00:02,836-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [70bbb93e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:00:02,841-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [42dee60b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:00:02,847-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [42dee60b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 61a4f432
>2018-05-30 00:00:02,898-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e3e8b41] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:00:02,898-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7e3e8b41] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7143fa8
>2018-05-30 00:00:02,902-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e3e8b41] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:00:02,903-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7e3e8b41] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:00:02,906-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e3e8b41] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:00:02,907-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7e3e8b41] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:00:02,911-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e3e8b41] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:00:02,911-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7e3e8b41] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:00:02,914-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7e3e8b41] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:00:02,916-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e3e8b41] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7e3e8b41, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:00:02,923-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2512470a
>2018-05-30 00:00:02,923-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:00:03,018-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:00:03,018-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:00:03,018-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:00:03,018-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:00:03,018-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2512470a
>2018-05-30 00:00:03,018-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:00:03,031-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [78fa3c14] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:00:03,032-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78fa3c14] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5777525e
>2018-05-30 00:00:03,034-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78fa3c14] FINISH, SetVdsStatusVDSCommand, log id: 5777525e
>2018-05-30 00:00:03,043-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fa3c14] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 78fa3c14, Job ID: f0269ad3-f77d-4b39-a2be-94dc40ae9abb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:00:03,047-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fa3c14] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 45193dca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:00:03,049-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fa3c14] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 45193dca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:00:03,060-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [28b7a34a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:00:03,061-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [28b7a34a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:00:04,221-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [42dee60b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:00:04,221-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [42dee60b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 61a4f432
>2018-05-30 00:00:04,225-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [42dee60b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:00:04,226-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [42dee60b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:00:04,229-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [42dee60b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:00:04,230-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [42dee60b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:00:04,234-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [42dee60b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:00:04,234-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [42dee60b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:00:04,238-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [42dee60b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:00:04,241-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [42dee60b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 42dee60b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:00:04,244-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7461c076
>2018-05-30 00:00:04,244-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:00:04,344-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:00:04,344-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:00:04,344-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:00:04,344-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:00:04,344-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7461c076
>2018-05-30 00:00:04,344-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:00:04,357-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [3c2a3c33] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:00:04,358-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3c2a3c33] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 380f685
>2018-05-30 00:00:04,360-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3c2a3c33] FINISH, SetVdsStatusVDSCommand, log id: 380f685
>2018-05-30 00:00:04,366-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3c2a3c33] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c2a3c33, Job ID: ae47bf9e-954b-4ef3-a000-f1a613be23ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:00:04,371-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3c2a3c33] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 70bbb93e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:00:04,373-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3c2a3c33] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 70bbb93e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:00:04,384-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [51ceeeca] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:00:04,385-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [51ceeeca] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [35459b94] Autorecovering 2 hosts
>2018-05-30 00:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [35459b94] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:05:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1239332f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:05:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1239332f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:05:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1239332f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:05:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1239332f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:05:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1239332f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 318964c4
>2018-05-30 00:05:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1239332f] FINISH, SetVdsStatusVDSCommand, log id: 318964c4
>2018-05-30 00:05:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1239332f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:05:00,073-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1239332f] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 1239332f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-30 00:05:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1239332f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:05:00,073-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1239332f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:05:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2bc1a894] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:05:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2bc1a894] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:05:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2bc1a894] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:05:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2bc1a894] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:05:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2bc1a894] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6eec63d6
>2018-05-30 00:05:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2bc1a894] FINISH, SetVdsStatusVDSCommand, log id: 6eec63d6
>2018-05-30 00:05:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [2bc1a894] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 652e3909
>2018-05-30 00:05:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [2bc1a894] FINISH, SetHaMaintenanceModeVDSCommand, log id: 652e3909
>2018-05-30 00:05:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2bc1a894] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:05:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2bc1a894] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:05:00,757-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [22ccdbc6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2d6db281
>2018-05-30 00:05:00,759-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [22ccdbc6] FINISH, GetHardwareInfoVDSCommand, log id: 2d6db281
>2018-05-30 00:05:00,772-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [1dfd46cf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:05:00,780-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [35d6ac0a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:05:00,786-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [262d44b7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:05:00,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [262d44b7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1773ed6
>2018-05-30 00:05:01,186-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [262d44b7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:05:01,186-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [262d44b7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1773ed6
>2018-05-30 00:05:01,190-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [262d44b7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:05:01,191-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [262d44b7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:05:01,195-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [262d44b7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:05:01,195-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [262d44b7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:05:01,199-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [262d44b7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:05:01,200-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [262d44b7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:05:01,203-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [262d44b7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:05:01,205-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [262d44b7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 262d44b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:05:01,212-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 126d130
>2018-05-30 00:05:01,212-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:05:01,312-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:05:01,312-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:05:01,312-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:05:01,312-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:05:01,312-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 126d130
>2018-05-30 00:05:01,312-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:05:01,325-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [480bb623] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:05:01,326-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [480bb623] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c74ed33
>2018-05-30 00:05:01,328-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [480bb623] FINISH, SetVdsStatusVDSCommand, log id: 2c74ed33
>2018-05-30 00:05:01,361-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [480bb623] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 480bb623, Job ID: 4d54ceac-2d6b-4541-9342-e8b7730d20a7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:05:01,365-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [480bb623] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 35d6ac0a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:05:01,368-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [480bb623] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 35d6ac0a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:05:01,378-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [34ec22bc] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:05:01,379-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [34ec22bc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:05:02,074-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [25ef22cb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 134ae8d1
>2018-05-30 00:05:02,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [25ef22cb] FINISH, GetHardwareInfoVDSCommand, log id: 134ae8d1
>2018-05-30 00:05:02,087-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [26e8e9b1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:05:02,092-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [6ca9db5f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:05:02,097-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [bccc1a4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:05:02,104-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [bccc1a4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2b549daa
>2018-05-30 00:05:02,478-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bccc1a4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:05:02,478-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [bccc1a4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2b549daa
>2018-05-30 00:05:02,482-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bccc1a4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:05:02,483-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [bccc1a4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:05:02,487-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bccc1a4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:05:02,487-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [bccc1a4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:05:02,491-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bccc1a4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:05:02,492-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [bccc1a4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:05:02,496-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [bccc1a4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:05:02,498-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bccc1a4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: bccc1a4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:05:02,501-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6f3d4310
>2018-05-30 00:05:02,501-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:05:02,600-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:05:02,600-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:05:02,600-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:05:02,600-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:05:02,600-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6f3d4310
>2018-05-30 00:05:02,600-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:05:02,614-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [2603ada0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:05:02,615-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2603ada0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a3a1f1e
>2018-05-30 00:05:02,617-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2603ada0] FINISH, SetVdsStatusVDSCommand, log id: 5a3a1f1e
>2018-05-30 00:05:02,648-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2603ada0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2603ada0, Job ID: 8395d0c3-8da2-4dfe-ba01-a81edac744d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:05:02,653-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2603ada0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6ca9db5f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:05:02,655-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2603ada0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6ca9db5f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:05:02,665-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [525e4361] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:05:02,666-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [525e4361] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:10:00,016-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5b27ba21] Autorecovering 2 hosts
>2018-05-30 00:10:00,016-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5b27ba21] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:10:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50c5fdc6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:10:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50c5fdc6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:10:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50c5fdc6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:10:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50c5fdc6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:10:00,055-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [50c5fdc6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4462c083
>2018-05-30 00:10:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [50c5fdc6] FINISH, SetVdsStatusVDSCommand, log id: 4462c083
>2018-05-30 00:10:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50c5fdc6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:10:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50c5fdc6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:10:00,062-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [50c5fdc6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:10:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5ebb9e62] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:10:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5ebb9e62] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:10:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5ebb9e62] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:10:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5ebb9e62] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:10:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5ebb9e62] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 93fbd56
>2018-05-30 00:10:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5ebb9e62] FINISH, SetVdsStatusVDSCommand, log id: 93fbd56
>2018-05-30 00:10:00,082-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [5ebb9e62] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7d3e1c3b
>2018-05-30 00:10:00,084-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [5ebb9e62] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7d3e1c3b
>2018-05-30 00:10:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5ebb9e62] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:10:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5ebb9e62] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:10:00,562-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6b3a9c50] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 42dfaec4
>2018-05-30 00:10:00,564-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6b3a9c50] FINISH, GetHardwareInfoVDSCommand, log id: 42dfaec4
>2018-05-30 00:10:00,578-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [76159a77] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:10:00,587-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [25811c05] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:10:00,593-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [27bd1513] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:10:00,652-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [27bd1513] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 43a292bf
>2018-05-30 00:10:01,028-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27bd1513] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:10:01,029-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [27bd1513] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 43a292bf
>2018-05-30 00:10:01,033-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27bd1513] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:10:01,033-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [27bd1513] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:10:01,037-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27bd1513] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:10:01,038-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [27bd1513] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:10:01,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27bd1513] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:10:01,043-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [27bd1513] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:10:01,046-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [27bd1513] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:10:01,049-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27bd1513] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 27bd1513, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:10:01,055-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6423b275
>2018-05-30 00:10:01,056-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:10:01,155-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:10:01,155-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:10:01,155-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:10:01,155-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:10:01,155-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6423b275
>2018-05-30 00:10:01,155-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:10:01,168-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [2f77c33] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:10:01,170-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2f77c33] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30f6d4cc
>2018-05-30 00:10:01,172-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2f77c33] FINISH, SetVdsStatusVDSCommand, log id: 30f6d4cc
>2018-05-30 00:10:01,180-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f77c33] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2f77c33, Job ID: 4ac694e7-29fd-4fe6-b8d3-4fd419b25152, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:10:01,185-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f77c33] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 25811c05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:10:01,187-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f77c33] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 25811c05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:10:01,197-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [338b2f35] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:10:01,198-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [338b2f35] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:10:02,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [2c1cc807] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 9023de0
>2018-05-30 00:10:02,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [2c1cc807] FINISH, GetHardwareInfoVDSCommand, log id: 9023de0
>2018-05-30 00:10:02,091-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [68df0059] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:10:02,096-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2872ba8c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:10:02,101-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4c30bff1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:10:02,109-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4c30bff1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6363c3a1
>2018-05-30 00:10:02,486-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c30bff1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:10:02,487-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4c30bff1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6363c3a1
>2018-05-30 00:10:02,491-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c30bff1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:10:02,492-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4c30bff1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:10:02,495-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c30bff1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:10:02,496-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4c30bff1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:10:02,500-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c30bff1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:10:02,501-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4c30bff1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:10:02,505-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4c30bff1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:10:02,508-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c30bff1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4c30bff1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:10:02,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 36108025
>2018-05-30 00:10:02,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:10:02,610-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:10:02,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:10:02,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:10:02,610-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:10:02,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 36108025
>2018-05-30 00:10:02,610-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:10:02,624-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [449efaac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:10:02,626-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [449efaac] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f89c730
>2018-05-30 00:10:02,628-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [449efaac] FINISH, SetVdsStatusVDSCommand, log id: 7f89c730
>2018-05-30 00:10:02,636-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [449efaac] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 449efaac, Job ID: 20ae1e6a-89db-4aba-84c9-e8854a5413cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:10:02,641-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [449efaac] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2872ba8c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:10:02,644-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [449efaac] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2872ba8c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:10:02,654-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [7a783d66] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:10:02,655-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [7a783d66] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7b784087] Autorecovering 2 hosts
>2018-05-30 00:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7b784087] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:15:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [191098e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:15:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [191098e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [191098e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [191098e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:15:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [191098e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76cbed8
>2018-05-30 00:15:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [191098e] FINISH, SetVdsStatusVDSCommand, log id: 76cbed8
>2018-05-30 00:15:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [191098e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:15:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [191098e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:15:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [191098e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:15:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5a266596] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5a266596] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5a266596] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5a266596] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:15:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5a266596] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 266d5d1a
>2018-05-30 00:15:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5a266596] FINISH, SetVdsStatusVDSCommand, log id: 266d5d1a
>2018-05-30 00:15:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5a266596] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 72419ff2
>2018-05-30 00:15:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5a266596] FINISH, SetHaMaintenanceModeVDSCommand, log id: 72419ff2
>2018-05-30 00:15:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5a266596] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:15:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5a266596] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:15:00,309-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [326eb818] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6bbce88c
>2018-05-30 00:15:00,310-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [326eb818] FINISH, GetHardwareInfoVDSCommand, log id: 6bbce88c
>2018-05-30 00:15:00,324-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [2b9524ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:15:00,332-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [461f4b8f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:15:00,337-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2105ed07] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:15:00,348-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2105ed07] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2ce6d997
>2018-05-30 00:15:00,730-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2105ed07] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:15:00,730-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2105ed07] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2ce6d997
>2018-05-30 00:15:00,734-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2105ed07] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:15:00,735-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2105ed07] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:15:00,739-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2105ed07] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:15:00,740-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2105ed07] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:15:00,744-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2105ed07] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:15:00,744-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2105ed07] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:15:00,747-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2105ed07] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:15:00,750-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2105ed07] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2105ed07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:15:00,756-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1aa19b65
>2018-05-30 00:15:00,756-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:15:00,854-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:15:00,854-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:15:00,854-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:15:00,854-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:15:00,854-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1aa19b65
>2018-05-30 00:15:00,854-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:15:00,868-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [5180e43a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:15:00,870-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5180e43a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 79c7352b
>2018-05-30 00:15:00,872-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5180e43a] FINISH, SetVdsStatusVDSCommand, log id: 79c7352b
>2018-05-30 00:15:00,904-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5180e43a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5180e43a, Job ID: 7dc966aa-099f-4384-b1bd-29aae80a5a81, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:15:00,909-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5180e43a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 461f4b8f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:15:00,911-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5180e43a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 461f4b8f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:15:00,923-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [416f6f08] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:15:00,924-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [416f6f08] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:15:01,968-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [55b44d22] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 243b5ec1
>2018-05-30 00:15:01,969-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [55b44d22] FINISH, GetHardwareInfoVDSCommand, log id: 243b5ec1
>2018-05-30 00:15:01,984-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [13ac7f2e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:15:01,989-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [681bd019] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:15:01,994-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [eb9eb2d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:15:02,002-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [eb9eb2d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 247db37c
>2018-05-30 00:15:02,375-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [eb9eb2d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:15:02,375-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [eb9eb2d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 247db37c
>2018-05-30 00:15:02,380-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [eb9eb2d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:15:02,380-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [eb9eb2d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:15:02,384-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [eb9eb2d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:15:02,385-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [eb9eb2d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:15:02,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [eb9eb2d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:15:02,389-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [eb9eb2d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:15:02,393-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [eb9eb2d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:15:02,395-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [eb9eb2d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: eb9eb2d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:15:02,402-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 48d538fc
>2018-05-30 00:15:02,402-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:15:02,499-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:15:02,499-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:15:02,499-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:15:02,499-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:15:02,499-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 48d538fc
>2018-05-30 00:15:02,499-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:15:02,513-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [4d7f4b58] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:15:02,515-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4d7f4b58] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f2edace
>2018-05-30 00:15:02,517-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4d7f4b58] FINISH, SetVdsStatusVDSCommand, log id: 1f2edace
>2018-05-30 00:15:02,552-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4d7f4b58] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4d7f4b58, Job ID: 4e6073b9-dc2e-4556-afe3-0a5d2942bbd1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:15:02,557-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4d7f4b58] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 681bd019, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:15:02,560-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4d7f4b58] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 681bd019, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:15:02,572-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [8442de3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:15:02,573-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [8442de3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:20:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [44db1c49] Autorecovering 2 hosts
>2018-05-30 00:20:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [44db1c49] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:20:00,012-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [763a39a4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:20:00,018-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [763a39a4] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:20:00,018-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [763a39a4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:20:00,018-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [763a39a4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:20:00,020-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [763a39a4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1fab73dd
>2018-05-30 00:20:00,022-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [763a39a4] FINISH, SetVdsStatusVDSCommand, log id: 1fab73dd
>2018-05-30 00:20:00,024-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [763a39a4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:20:00,025-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [763a39a4] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:20:00,025-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [763a39a4] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:20:00,030-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7062d10] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:20:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7062d10] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:20:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7062d10] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:20:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7062d10] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:20:00,038-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7062d10] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6854351f
>2018-05-30 00:20:00,040-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7062d10] FINISH, SetVdsStatusVDSCommand, log id: 6854351f
>2018-05-30 00:20:00,043-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7062d10] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4137d34b
>2018-05-30 00:20:00,045-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7062d10] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4137d34b
>2018-05-30 00:20:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7062d10] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:20:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7062d10] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:20:00,213-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4d7f55fc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 686d6d0d
>2018-05-30 00:20:00,215-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4d7f55fc] FINISH, GetHardwareInfoVDSCommand, log id: 686d6d0d
>2018-05-30 00:20:00,229-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6a60236d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:20:00,233-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [2eab8ad6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:20:00,238-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2d3c1ef7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:20:00,245-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2d3c1ef7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1bb0538e
>2018-05-30 00:20:00,623-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2d3c1ef7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:20:00,623-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2d3c1ef7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1bb0538e
>2018-05-30 00:20:00,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2d3c1ef7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:20:00,628-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2d3c1ef7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:20:00,632-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2d3c1ef7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:20:00,633-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2d3c1ef7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:20:00,637-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2d3c1ef7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:20:00,638-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2d3c1ef7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:20:00,641-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2d3c1ef7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:20:00,644-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2d3c1ef7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2d3c1ef7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:20:00,647-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6b224142
>2018-05-30 00:20:00,647-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:20:00,747-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:20:00,747-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:20:00,747-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:20:00,747-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:20:00,747-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6b224142
>2018-05-30 00:20:00,747-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:20:00,761-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [581f905] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:20:00,762-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [581f905] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5888ebd6
>2018-05-30 00:20:00,764-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [581f905] FINISH, SetVdsStatusVDSCommand, log id: 5888ebd6
>2018-05-30 00:20:00,770-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [581f905] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 581f905, Job ID: ec93d87e-191a-4080-8c99-78998e1055d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:20:00,774-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [581f905] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2eab8ad6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:20:00,776-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [581f905] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2eab8ad6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:20:00,788-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [7a95c669] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:20:00,789-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7a95c669] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:20:01,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [5169a91f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 519a2105
>2018-05-30 00:20:01,536-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [5169a91f] FINISH, GetHardwareInfoVDSCommand, log id: 519a2105
>2018-05-30 00:20:01,549-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [71a9da51] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:20:01,553-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [1d4931d9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:20:01,559-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [147f90f9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:20:01,566-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [147f90f9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: bd6ae0
>2018-05-30 00:20:01,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [147f90f9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:20:01,938-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [147f90f9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: bd6ae0
>2018-05-30 00:20:01,942-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [147f90f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:20:01,943-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [147f90f9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:20:01,947-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [147f90f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:20:01,948-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [147f90f9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:20:01,951-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [147f90f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:20:01,952-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [147f90f9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:20:01,955-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [147f90f9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:20:01,958-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [147f90f9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 147f90f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:20:01,960-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 58597787
>2018-05-30 00:20:01,960-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:20:02,059-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:20:02,059-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:20:02,059-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:20:02,059-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:20:02,059-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 58597787
>2018-05-30 00:20:02,059-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:20:02,072-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [6219d895] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:20:02,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6219d895] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 660da0f2
>2018-05-30 00:20:02,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6219d895] FINISH, SetVdsStatusVDSCommand, log id: 660da0f2
>2018-05-30 00:20:02,081-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6219d895] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6219d895, Job ID: 9faf1465-8243-4986-8717-17b4632f8b09, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:20:02,086-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6219d895] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d4931d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:20:02,088-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6219d895] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d4931d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:20:02,099-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [1e81fa54] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:20:02,100-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [1e81fa54] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2c885480] Autorecovering 2 hosts
>2018-05-30 00:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2c885480] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:25:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61c81534] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:25:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61c81534] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:25:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61c81534] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:25:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61c81534] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:25:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [61c81534] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d4314da
>2018-05-30 00:25:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [61c81534] FINISH, SetVdsStatusVDSCommand, log id: 3d4314da
>2018-05-30 00:25:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61c81534] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:25:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61c81534] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:25:00,073-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [61c81534] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:25:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6248f2e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:25:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6248f2e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:25:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6248f2e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:25:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6248f2e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:25:00,086-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6248f2e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b1d1075
>2018-05-30 00:25:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6248f2e] FINISH, SetVdsStatusVDSCommand, log id: 4b1d1075
>2018-05-30 00:25:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6248f2e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 455dd458
>2018-05-30 00:25:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6248f2e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 455dd458
>2018-05-30 00:25:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6248f2e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:25:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6248f2e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:25:01,477-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [486c90fb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3277b072
>2018-05-30 00:25:01,479-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [486c90fb] FINISH, GetHardwareInfoVDSCommand, log id: 3277b072
>2018-05-30 00:25:01,493-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [474eb013] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:25:01,501-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [61f02d6c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:25:01,506-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7a918aec] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:25:01,559-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7a918aec] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1383963f
>2018-05-30 00:25:01,922-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7a918aec] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:25:01,922-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7a918aec] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1383963f
>2018-05-30 00:25:01,926-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7a918aec] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:25:01,927-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7a918aec] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:25:01,931-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7a918aec] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:25:01,932-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7a918aec] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:25:01,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7a918aec] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:25:01,936-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7a918aec] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:25:01,939-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7a918aec] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:25:01,942-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7a918aec] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7a918aec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:25:01,948-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5d984f23
>2018-05-30 00:25:01,948-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:25:02,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:25:02,042-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:25:02,042-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:25:02,042-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:25:02,042-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5d984f23
>2018-05-30 00:25:02,042-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:25:02,056-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [6bf6c7d2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:25:02,057-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6bf6c7d2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 20bf61c2
>2018-05-30 00:25:02,060-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6bf6c7d2] FINISH, SetVdsStatusVDSCommand, log id: 20bf61c2
>2018-05-30 00:25:02,068-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bf6c7d2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6bf6c7d2, Job ID: 32c9e7f5-23d0-4036-bd9b-c2ca6b227070, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:25:02,072-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bf6c7d2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 61f02d6c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:25:02,075-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bf6c7d2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 61f02d6c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:25:02,085-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [a385649] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:25:02,086-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [a385649] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:25:02,676-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [42f235fd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6dc32606
>2018-05-30 00:25:02,678-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [42f235fd] FINISH, GetHardwareInfoVDSCommand, log id: 6dc32606
>2018-05-30 00:25:02,691-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [559fc9a3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:25:02,695-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [269bb070] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:25:02,700-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7036138] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:25:02,712-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7036138] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 38a84f8a
>2018-05-30 00:25:03,095-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7036138] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:25:03,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7036138] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 38a84f8a
>2018-05-30 00:25:03,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7036138] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:25:03,100-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7036138] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:25:03,103-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7036138] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:25:03,104-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7036138] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:25:03,108-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7036138] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:25:03,108-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7036138] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:25:03,111-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7036138] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:25:03,114-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7036138] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7036138, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:25:03,120-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3f2ceea9
>2018-05-30 00:25:03,120-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:25:03,214-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:25:03,214-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:25:03,214-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:25:03,214-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:25:03,214-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3f2ceea9
>2018-05-30 00:25:03,214-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:25:03,227-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [408f65a4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:25:03,228-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [408f65a4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42947d18
>2018-05-30 00:25:03,230-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [408f65a4] FINISH, SetVdsStatusVDSCommand, log id: 42947d18
>2018-05-30 00:25:03,239-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [408f65a4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 408f65a4, Job ID: 295ad171-6da0-4cc7-b9d9-b7eda243b883, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:25:03,244-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [408f65a4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 269bb070, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:25:03,246-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [408f65a4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 269bb070, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:25:03,257-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [17e2c355] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:25:03,258-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [17e2c355] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [27f2e42b] Autorecovering 2 hosts
>2018-05-30 00:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [27f2e42b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:30:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7ae8c1e9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:30:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7ae8c1e9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:30:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7ae8c1e9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:30:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7ae8c1e9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:30:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7ae8c1e9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 593aa642
>2018-05-30 00:30:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7ae8c1e9] FINISH, SetVdsStatusVDSCommand, log id: 593aa642
>2018-05-30 00:30:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7ae8c1e9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:30:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7ae8c1e9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:30:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7ae8c1e9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:30:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [412db4c7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:30:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [412db4c7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:30:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [412db4c7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:30:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [412db4c7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:30:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [412db4c7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 725b2173
>2018-05-30 00:30:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [412db4c7] FINISH, SetVdsStatusVDSCommand, log id: 725b2173
>2018-05-30 00:30:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [412db4c7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4c5465ea
>2018-05-30 00:30:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [412db4c7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4c5465ea
>2018-05-30 00:30:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [412db4c7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:30:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [412db4c7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:30:00,927-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [466f4a46] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2a290f98
>2018-05-30 00:30:00,929-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [466f4a46] FINISH, GetHardwareInfoVDSCommand, log id: 2a290f98
>2018-05-30 00:30:00,943-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [28defb34] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:30:00,952-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [416821ec] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:30:00,957-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [47e22dd8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:30:00,969-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [47e22dd8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6cd03b0b
>2018-05-30 00:30:01,347-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [47e22dd8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:30:01,347-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [47e22dd8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6cd03b0b
>2018-05-30 00:30:01,351-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [47e22dd8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:30:01,351-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [47e22dd8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:30:01,355-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [47e22dd8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:30:01,356-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [47e22dd8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:30:01,360-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [47e22dd8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:30:01,360-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [47e22dd8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:30:01,363-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [47e22dd8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:30:01,366-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [47e22dd8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 47e22dd8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:30:01,373-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2efd39da
>2018-05-30 00:30:01,373-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:30:01,472-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:30:01,472-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:30:01,472-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:30:01,472-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:30:01,472-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2efd39da
>2018-05-30 00:30:01,472-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:30:01,487-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [5f1de0b0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:30:01,488-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5f1de0b0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31ccf66f
>2018-05-30 00:30:01,491-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5f1de0b0] FINISH, SetVdsStatusVDSCommand, log id: 31ccf66f
>2018-05-30 00:30:01,523-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5f1de0b0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5f1de0b0, Job ID: 8f957041-a59e-4a88-9a7d-826868d6b4d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:30:01,528-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5f1de0b0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 416821ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:30:01,530-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5f1de0b0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 416821ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:30:01,541-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [15a71089] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:30:01,543-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [15a71089] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:30:02,723-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3afbf1a2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5d051460
>2018-05-30 00:30:02,725-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3afbf1a2] FINISH, GetHardwareInfoVDSCommand, log id: 5d051460
>2018-05-30 00:30:02,739-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [1db12efd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:30:02,743-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [29c8039f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:30:02,748-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1087d5b2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:30:02,755-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1087d5b2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4695807d
>2018-05-30 00:30:03,120-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1087d5b2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:30:03,120-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1087d5b2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4695807d
>2018-05-30 00:30:03,124-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1087d5b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:30:03,125-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1087d5b2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:30:03,129-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1087d5b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:30:03,129-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1087d5b2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:30:03,133-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1087d5b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:30:03,134-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1087d5b2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:30:03,136-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1087d5b2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:30:03,139-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1087d5b2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1087d5b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:30:03,142-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4edf40d6
>2018-05-30 00:30:03,142-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:30:03,243-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:30:03,243-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:30:03,243-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:30:03,243-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:30:03,243-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4edf40d6
>2018-05-30 00:30:03,243-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:30:03,256-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [6567c576] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:30:03,258-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6567c576] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 793e106b
>2018-05-30 00:30:03,260-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6567c576] FINISH, SetVdsStatusVDSCommand, log id: 793e106b
>2018-05-30 00:30:03,266-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6567c576] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6567c576, Job ID: 100edb10-be06-42c8-96e5-023738eda080, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:30:03,270-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6567c576] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29c8039f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:30:03,273-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6567c576] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29c8039f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:30:03,284-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [7a8a0056] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:30:03,285-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7a8a0056] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:35:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [350b7880] Autorecovering 2 hosts
>2018-05-30 00:35:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [350b7880] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:35:00,011-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7d71fd62] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:35:00,017-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7d71fd62] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:35:00,017-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7d71fd62] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:35:00,017-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7d71fd62] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:35:00,019-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7d71fd62] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d0ed533
>2018-05-30 00:35:00,021-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7d71fd62] FINISH, SetVdsStatusVDSCommand, log id: d0ed533
>2018-05-30 00:35:00,022-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7d71fd62] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:35:00,023-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7d71fd62] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:35:00,023-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7d71fd62] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:35:00,029-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [62fb5d13] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:35:00,035-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [62fb5d13] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:35:00,035-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [62fb5d13] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:35:00,035-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [62fb5d13] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:35:00,036-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [62fb5d13] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3413ce54
>2018-05-30 00:35:00,038-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [62fb5d13] FINISH, SetVdsStatusVDSCommand, log id: 3413ce54
>2018-05-30 00:35:00,041-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [62fb5d13] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 23113d83
>2018-05-30 00:35:00,043-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [62fb5d13] FINISH, SetHaMaintenanceModeVDSCommand, log id: 23113d83
>2018-05-30 00:35:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [62fb5d13] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:35:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [62fb5d13] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:35:00,894-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [145694a5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6a9d2606
>2018-05-30 00:35:00,896-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [145694a5] FINISH, GetHardwareInfoVDSCommand, log id: 6a9d2606
>2018-05-30 00:35:00,908-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [28c4cd84] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:35:00,912-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [17c22e11] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:35:00,938-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [c948807] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:35:00,946-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [c948807] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 73f1ca1b
>2018-05-30 00:35:02,192-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [10746a0e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 78a46556
>2018-05-30 00:35:02,194-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [10746a0e] FINISH, GetHardwareInfoVDSCommand, log id: 78a46556
>2018-05-30 00:35:02,208-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [427b00c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:35:02,213-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [2642136f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:35:02,218-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6d9fe52] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:35:02,225-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6d9fe52] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3dd3f382
>2018-05-30 00:35:02,313-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [c948807] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:35:02,313-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [c948807] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 73f1ca1b
>2018-05-30 00:35:02,317-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [c948807] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:35:02,318-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [c948807] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:35:02,321-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [c948807] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:35:02,322-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [c948807] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:35:02,325-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [c948807] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:35:02,326-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [c948807] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:35:02,329-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [c948807] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:35:02,331-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [c948807] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: c948807, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:35:02,334-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2e59b44a
>2018-05-30 00:35:02,334-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:35:02,430-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:35:02,430-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:35:02,430-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:35:02,430-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:35:02,430-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2e59b44a
>2018-05-30 00:35:02,430-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:35:02,444-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [156d020] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:35:02,445-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [156d020] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 57938522
>2018-05-30 00:35:02,447-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [156d020] FINISH, SetVdsStatusVDSCommand, log id: 57938522
>2018-05-30 00:35:02,476-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [156d020] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 156d020, Job ID: 587d27d3-1b07-4a81-a207-a342815391a2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:35:02,481-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [156d020] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17c22e11, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:35:02,483-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [156d020] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17c22e11, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:35:02,494-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [2b1bf34a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:35:02,495-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [2b1bf34a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:35:02,601-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6d9fe52] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:35:02,602-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6d9fe52] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3dd3f382
>2018-05-30 00:35:02,605-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6d9fe52] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:35:02,606-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6d9fe52] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:35:02,610-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6d9fe52] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:35:02,610-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6d9fe52] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:35:02,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6d9fe52] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:35:02,615-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6d9fe52] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:35:02,618-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6d9fe52] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:35:02,621-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6d9fe52] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6d9fe52, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:35:02,624-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 28d2c38d
>2018-05-30 00:35:02,624-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:35:02,721-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:35:02,721-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:35:02,721-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:35:02,721-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:35:02,721-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 28d2c38d
>2018-05-30 00:35:02,721-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:35:02,734-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [22f6c3fa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:35:02,736-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [22f6c3fa] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 37e51f99
>2018-05-30 00:35:02,738-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [22f6c3fa] FINISH, SetVdsStatusVDSCommand, log id: 37e51f99
>2018-05-30 00:35:02,743-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [22f6c3fa] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22f6c3fa, Job ID: e58712f2-5543-4709-9709-e09176bfe389, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:35:02,748-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [22f6c3fa] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2642136f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:35:02,750-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [22f6c3fa] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2642136f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:35:02,761-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [5f679eb2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:35:02,762-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [5f679eb2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2026d6e] Autorecovering 2 hosts
>2018-05-30 00:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2026d6e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:40:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4524eb5b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4524eb5b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4524eb5b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4524eb5b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:40:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4524eb5b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: eebc68a
>2018-05-30 00:40:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4524eb5b] FINISH, SetVdsStatusVDSCommand, log id: eebc68a
>2018-05-30 00:40:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4524eb5b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:40:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4524eb5b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:40:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4524eb5b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:40:00,131-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [56c23ea2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:40:00,137-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [56c23ea2] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:40:00,137-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [56c23ea2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:40:00,137-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [56c23ea2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:40:00,139-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [56c23ea2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4056ef39
>2018-05-30 00:40:00,141-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [56c23ea2] FINISH, SetVdsStatusVDSCommand, log id: 4056ef39
>2018-05-30 00:40:00,144-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [56c23ea2] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5f74914
>2018-05-30 00:40:00,146-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [56c23ea2] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5f74914
>2018-05-30 00:40:00,146-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [56c23ea2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:40:00,147-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [56c23ea2] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:40:00,341-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [1e9a3a59] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 23c14e0e
>2018-05-30 00:40:00,343-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [1e9a3a59] FINISH, GetHardwareInfoVDSCommand, log id: 23c14e0e
>2018-05-30 00:40:00,355-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [58c86255] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:40:00,364-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [2153b896] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:40:00,369-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [24a3c2e3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:40:00,381-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [24a3c2e3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1438d1ed
>2018-05-30 00:40:00,759-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [24a3c2e3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:40:00,759-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [24a3c2e3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1438d1ed
>2018-05-30 00:40:00,763-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [24a3c2e3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:40:00,764-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [24a3c2e3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:40:00,768-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [24a3c2e3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:40:00,768-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [24a3c2e3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:40:00,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [24a3c2e3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:40:00,773-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [24a3c2e3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:40:00,776-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [24a3c2e3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:40:00,779-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [24a3c2e3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 24a3c2e3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:40:00,786-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6c295148
>2018-05-30 00:40:00,786-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:40:00,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:40:00,881-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:40:00,881-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:40:00,881-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:40:00,881-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6c295148
>2018-05-30 00:40:00,881-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:40:00,895-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [577becbc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:40:00,896-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [577becbc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19036db5
>2018-05-30 00:40:00,898-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [577becbc] FINISH, SetVdsStatusVDSCommand, log id: 19036db5
>2018-05-30 00:40:00,930-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [577becbc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 577becbc, Job ID: 2682374d-e062-460a-a9ee-494cebd6f81b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:40:00,934-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [577becbc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2153b896, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:40:00,936-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [577becbc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2153b896, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:40:00,948-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [669697fe] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:40:00,949-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [669697fe] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:40:03,205-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [505c6afd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 60e27f89
>2018-05-30 00:40:03,206-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [505c6afd] FINISH, GetHardwareInfoVDSCommand, log id: 60e27f89
>2018-05-30 00:40:03,218-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [22a15ce9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:40:03,227-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [3b12692a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:40:03,231-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [27f6dcaf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:40:03,243-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [27f6dcaf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 65640423
>2018-05-30 00:40:03,625-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27f6dcaf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:40:03,625-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [27f6dcaf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 65640423
>2018-05-30 00:40:03,629-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27f6dcaf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:40:03,630-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [27f6dcaf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:40:03,634-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27f6dcaf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:40:03,635-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [27f6dcaf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:40:03,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27f6dcaf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:40:03,639-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [27f6dcaf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:40:03,642-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [27f6dcaf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:40:03,644-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27f6dcaf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 27f6dcaf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:40:03,651-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6f1fa9eb
>2018-05-30 00:40:03,651-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:40:03,749-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:40:03,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:40:03,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:40:03,749-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:40:03,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6f1fa9eb
>2018-05-30 00:40:03,749-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:40:03,762-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [2acee891] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:40:03,763-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2acee891] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26c5a9c9
>2018-05-30 00:40:03,765-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2acee891] FINISH, SetVdsStatusVDSCommand, log id: 26c5a9c9
>2018-05-30 00:40:03,796-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2acee891] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2acee891, Job ID: bc2a7e53-84a1-4844-9c9a-1aa84d922295, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:40:03,801-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2acee891] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b12692a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:40:03,803-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2acee891] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b12692a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:40:03,814-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [633bed06] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:40:03,815-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [633bed06] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [4435b1e8] Autorecovering 2 hosts
>2018-05-30 00:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [4435b1e8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:45:00,032-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [10498b9a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:45:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [10498b9a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:45:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [10498b9a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:45:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [10498b9a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:45:00,040-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [10498b9a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7b6289f
>2018-05-30 00:45:00,043-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [10498b9a] FINISH, SetVdsStatusVDSCommand, log id: 7b6289f
>2018-05-30 00:45:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [10498b9a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:45:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [10498b9a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:45:00,046-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [10498b9a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:45:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [57494d52] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:45:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [57494d52] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:45:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [57494d52] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:45:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [57494d52] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:45:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [57494d52] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6655a3cb
>2018-05-30 00:45:00,062-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [57494d52] FINISH, SetVdsStatusVDSCommand, log id: 6655a3cb
>2018-05-30 00:45:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [57494d52] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1e8179f1
>2018-05-30 00:45:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [57494d52] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1e8179f1
>2018-05-30 00:45:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [57494d52] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [57494d52] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:45:01,533-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3e51262f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6a1ce103
>2018-05-30 00:45:01,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3e51262f] FINISH, GetHardwareInfoVDSCommand, log id: 6a1ce103
>2018-05-30 00:45:01,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [762b2f4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 447fc3fb
>2018-05-30 00:45:01,537-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [762b2f4] FINISH, GetHardwareInfoVDSCommand, log id: 447fc3fb
>2018-05-30 00:45:01,550-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [6db648f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:45:01,556-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7bc34fbd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:45:01,559-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [5f0dfff9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:45:01,564-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [65794301] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:45:01,566-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [6fac2791] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:45:01,571-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [6b104ac5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:45:01,573-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [65794301] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1fcf01ab
>2018-05-30 00:45:01,578-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [6b104ac5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6bc1f8ad
>2018-05-30 00:45:01,955-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6b104ac5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:45:01,955-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [6b104ac5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6bc1f8ad
>2018-05-30 00:45:01,959-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6b104ac5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:45:01,960-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6b104ac5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:45:01,963-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6b104ac5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:45:01,964-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6b104ac5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:45:01,968-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6b104ac5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:45:01,969-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6b104ac5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:45:01,972-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [6b104ac5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:45:01,974-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6b104ac5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6b104ac5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:45:01,977-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 587b0f42
>2018-05-30 00:45:01,977-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:45:02,071-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:45:02,071-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:45:02,071-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:45:02,071-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:45:02,071-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 587b0f42
>2018-05-30 00:45:02,071-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:45:02,085-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [70c1dfd3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:45:02,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [70c1dfd3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46a32337
>2018-05-30 00:45:02,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [70c1dfd3] FINISH, SetVdsStatusVDSCommand, log id: 46a32337
>2018-05-30 00:45:02,095-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [70c1dfd3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 70c1dfd3, Job ID: 8e3ade66-0108-4e48-abde-b6c23be53419, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:45:02,099-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [70c1dfd3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6fac2791, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:45:02,102-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [70c1dfd3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6fac2791, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:45:02,113-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [94aaa4a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:45:02,114-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [94aaa4a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:45:02,937-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65794301] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:45:02,937-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [65794301] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1fcf01ab
>2018-05-30 00:45:02,941-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65794301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:45:02,941-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [65794301] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:45:02,945-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65794301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:45:02,946-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [65794301] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:45:02,950-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65794301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:45:02,950-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [65794301] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:45:02,954-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [65794301] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:45:02,956-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65794301] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 65794301, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:45:02,959-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: ccfe6fb
>2018-05-30 00:45:02,959-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:45:03,055-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:45:03,055-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:45:03,055-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:45:03,055-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:45:03,055-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: ccfe6fb
>2018-05-30 00:45:03,055-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:45:03,068-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [3c8bf25c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:45:03,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3c8bf25c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 73ea559e
>2018-05-30 00:45:03,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3c8bf25c] FINISH, SetVdsStatusVDSCommand, log id: 73ea559e
>2018-05-30 00:45:03,077-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c8bf25c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c8bf25c, Job ID: 202f3eb9-3eee-49f5-9534-36650ac629b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:45:03,081-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c8bf25c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5f0dfff9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:45:03,083-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c8bf25c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5f0dfff9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:45:03,094-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [3bec7ca9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:45:03,095-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [3bec7ca9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:50:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6103baad] Autorecovering 2 hosts
>2018-05-30 00:50:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6103baad] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:50:00,030-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [745d87c1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:50:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [745d87c1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:50:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [745d87c1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:50:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [745d87c1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:50:00,039-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [745d87c1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1ce46562
>2018-05-30 00:50:00,041-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [745d87c1] FINISH, SetVdsStatusVDSCommand, log id: 1ce46562
>2018-05-30 00:50:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [745d87c1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:50:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [745d87c1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:50:00,043-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [745d87c1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:50:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [441b7a31] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:50:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [441b7a31] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:50:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [441b7a31] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:50:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [441b7a31] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:50:00,057-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [441b7a31] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19b5452
>2018-05-30 00:50:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [441b7a31] FINISH, SetVdsStatusVDSCommand, log id: 19b5452
>2018-05-30 00:50:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [441b7a31] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 60a8e005
>2018-05-30 00:50:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [441b7a31] FINISH, SetHaMaintenanceModeVDSCommand, log id: 60a8e005
>2018-05-30 00:50:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [441b7a31] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:50:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [441b7a31] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:50:00,780-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [493b3dc4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4f1638ed
>2018-05-30 00:50:00,781-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [493b3dc4] FINISH, GetHardwareInfoVDSCommand, log id: 4f1638ed
>2018-05-30 00:50:00,794-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [6b0752a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:50:00,799-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1f3bb1cb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:50:00,805-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6ee5e1e9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:50:00,816-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6ee5e1e9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 42dfd9cc
>2018-05-30 00:50:01,185-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6ee5e1e9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:50:01,185-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6ee5e1e9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 42dfd9cc
>2018-05-30 00:50:01,189-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6ee5e1e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:50:01,190-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6ee5e1e9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:50:01,193-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6ee5e1e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:50:01,194-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6ee5e1e9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:50:01,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6ee5e1e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:50:01,199-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6ee5e1e9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:50:01,203-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6ee5e1e9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:50:01,206-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6ee5e1e9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6ee5e1e9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:50:01,214-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1f9477c2
>2018-05-30 00:50:01,214-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:50:01,309-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:50:01,309-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:50:01,309-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:50:01,309-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:50:01,309-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1f9477c2
>2018-05-30 00:50:01,309-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:50:01,323-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [3b65d568] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:50:01,324-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3b65d568] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 84f289c
>2018-05-30 00:50:01,326-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3b65d568] FINISH, SetVdsStatusVDSCommand, log id: 84f289c
>2018-05-30 00:50:01,334-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3b65d568] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3b65d568, Job ID: 434e5584-bfb7-4fc2-9c11-accdf0464ca1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:50:01,339-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3b65d568] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1f3bb1cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:50:01,341-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3b65d568] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1f3bb1cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:50:01,352-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2dcbdebb] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:50:01,353-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2dcbdebb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:50:02,733-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3e51d283] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6a581112
>2018-05-30 00:50:02,735-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3e51d283] FINISH, GetHardwareInfoVDSCommand, log id: 6a581112
>2018-05-30 00:50:02,748-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [63ee8e66] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:50:02,753-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [8e182d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:50:02,758-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3e07120e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:50:02,765-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3e07120e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 27379b3f
>2018-05-30 00:50:03,148-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3e07120e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:50:03,148-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3e07120e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 27379b3f
>2018-05-30 00:50:03,152-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3e07120e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:50:03,153-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3e07120e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:50:03,156-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3e07120e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:50:03,157-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3e07120e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:50:03,161-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3e07120e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:50:03,162-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3e07120e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:50:03,164-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3e07120e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:50:03,166-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3e07120e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3e07120e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:50:03,169-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 74654907
>2018-05-30 00:50:03,169-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:50:03,268-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:50:03,268-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:50:03,268-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:50:03,268-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:50:03,268-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 74654907
>2018-05-30 00:50:03,268-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:50:03,280-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [3a15d3b6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:50:03,281-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3a15d3b6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 14a6cc9d
>2018-05-30 00:50:03,283-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3a15d3b6] FINISH, SetVdsStatusVDSCommand, log id: 14a6cc9d
>2018-05-30 00:50:03,291-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3a15d3b6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3a15d3b6, Job ID: 11353512-4cda-43f6-80c7-a3b77d825174, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:50:03,296-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3a15d3b6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 8e182d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:50:03,298-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3a15d3b6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 8e182d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:50:03,308-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [7208bff0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:50:03,309-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7208bff0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:54:43,448-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler5) [436482c5] Attempting to update VMs/Templates Ovf.
>2018-05-30 00:54:43,451-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 00:54:43,451-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 00:54:43,451-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:54:43,456-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-30 00:54:43,460-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-30 00:54:43,460-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Attempting to update template OVFs in Data Center 'Default'
>2018-05-30 00:54:43,460-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-30 00:54:43,460-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 00:54:43,460-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 00:54:43,461-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 00:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [20accf8d] Autorecovering 2 hosts
>2018-05-30 00:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [20accf8d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:55:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [9c48754] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:55:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [9c48754] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:55:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [9c48754] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:55:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [9c48754] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:55:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [9c48754] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43b6e58e
>2018-05-30 00:55:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [9c48754] FINISH, SetVdsStatusVDSCommand, log id: 43b6e58e
>2018-05-30 00:55:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [9c48754] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:55:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [9c48754] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 00:55:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [9c48754] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:55:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [66b43539] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:55:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [66b43539] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 00:55:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [66b43539] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:55:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [66b43539] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:55:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [66b43539] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7fae1e73
>2018-05-30 00:55:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [66b43539] FINISH, SetVdsStatusVDSCommand, log id: 7fae1e73
>2018-05-30 00:55:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [66b43539] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 18ab9f2d
>2018-05-30 00:55:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [66b43539] FINISH, SetHaMaintenanceModeVDSCommand, log id: 18ab9f2d
>2018-05-30 00:55:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [66b43539] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 00:55:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [66b43539] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 00:55:00,974-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [15f33d40] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 541eb4f0
>2018-05-30 00:55:00,976-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [15f33d40] FINISH, GetHardwareInfoVDSCommand, log id: 541eb4f0
>2018-05-30 00:55:00,989-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [77dfd55e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:55:00,997-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [20b30cf3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:55:01,002-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [65242552] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:55:01,014-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [65242552] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 850e8cb
>2018-05-30 00:55:01,397-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65242552] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:55:01,397-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [65242552] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 850e8cb
>2018-05-30 00:55:01,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65242552] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:55:01,402-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [65242552] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:55:01,406-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65242552] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:55:01,407-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [65242552] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:55:01,410-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65242552] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:55:01,411-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [65242552] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:55:01,414-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [65242552] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:55:01,417-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65242552] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 65242552, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:55:01,423-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 686bb331
>2018-05-30 00:55:01,423-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:55:01,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:55:01,520-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:55:01,520-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:55:01,520-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:55:01,520-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 686bb331
>2018-05-30 00:55:01,520-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:55:01,534-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [68e8754b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:55:01,535-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [68e8754b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f33aa5d
>2018-05-30 00:55:01,537-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [68e8754b] FINISH, SetVdsStatusVDSCommand, log id: 1f33aa5d
>2018-05-30 00:55:01,569-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [68e8754b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 68e8754b, Job ID: 743c618d-652c-446e-a79a-44609da1b052, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:55:01,574-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [68e8754b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 20b30cf3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:55:01,576-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [68e8754b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 20b30cf3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:55:01,587-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [6efc6cc2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 00:55:01,589-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [6efc6cc2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 00:55:02,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4d819e94] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 374582e9
>2018-05-30 00:55:02,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4d819e94] FINISH, GetHardwareInfoVDSCommand, log id: 374582e9
>2018-05-30 00:55:02,090-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [115eebac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:55:02,098-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1d1e6d8f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:55:02,103-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [39528918] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 00:55:02,114-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [39528918] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 64815155
>2018-05-30 00:55:02,486-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [39528918] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 00:55:02,486-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [39528918] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 64815155
>2018-05-30 00:55:02,490-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [39528918] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:55:02,491-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [39528918] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:55:02,494-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [39528918] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:55:02,495-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [39528918] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:55:02,499-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [39528918] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 00:55:02,500-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [39528918] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 00:55:02,502-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [39528918] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 00:55:02,505-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [39528918] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 39528918, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 00:55:02,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 13020a06
>2018-05-30 00:55:02,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 00:55:02,612-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:55:02,612-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 00:55:02,613-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 00:55:02,613-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 00:55:02,613-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 13020a06
>2018-05-30 00:55:02,613-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 00:55:02,627-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [4def9e8a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:55:02,628-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4def9e8a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2df0bcdd
>2018-05-30 00:55:02,630-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4def9e8a] FINISH, SetVdsStatusVDSCommand, log id: 2df0bcdd
>2018-05-30 00:55:02,636-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4def9e8a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4def9e8a, Job ID: 2d0d8e18-fa18-444a-871f-693f7f513c40, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 00:55:02,640-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4def9e8a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d1e6d8f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 00:55:02,642-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4def9e8a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d1e6d8f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 00:55:02,653-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [177d9cbf] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 00:55:02,654-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [177d9cbf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [6a229617] Autorecovering 2 hosts
>2018-05-30 01:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [6a229617] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:00:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c992ff4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c992ff4] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c992ff4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:00:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c992ff4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:00:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6c992ff4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1a22cd33
>2018-05-30 01:00:01,637-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6c992ff4] FINISH, SetVdsStatusVDSCommand, log id: 1a22cd33
>2018-05-30 01:00:01,639-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c992ff4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:00:01,640-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c992ff4] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:00:01,640-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [6c992ff4] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:00:01,647-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [78a3b7cc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:00:01,653-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [78a3b7cc] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:00:01,653-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [78a3b7cc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:00:01,653-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [78a3b7cc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:00:01,655-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [78a3b7cc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7ff2829b
>2018-05-30 01:00:01,657-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [78a3b7cc] FINISH, SetVdsStatusVDSCommand, log id: 7ff2829b
>2018-05-30 01:00:01,659-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [78a3b7cc] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 8d65a94
>2018-05-30 01:00:01,661-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [78a3b7cc] FINISH, SetHaMaintenanceModeVDSCommand, log id: 8d65a94
>2018-05-30 01:00:01,661-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [78a3b7cc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:00:01,663-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [78a3b7cc] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:00:02,235-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [9bae053] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5691f004
>2018-05-30 01:00:02,237-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [9bae053] FINISH, GetHardwareInfoVDSCommand, log id: 5691f004
>2018-05-30 01:00:02,249-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [4e466d88] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:00:02,254-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [33a4da62] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:00:02,259-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5da674f0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:00:02,265-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5da674f0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7a164128
>2018-05-30 01:00:02,637-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5da674f0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:00:02,637-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5da674f0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7a164128
>2018-05-30 01:00:02,641-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5da674f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:00:02,642-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5da674f0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:00:02,646-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5da674f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:00:02,647-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5da674f0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:00:02,650-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5da674f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:00:02,651-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5da674f0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:00:02,654-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5da674f0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:00:02,657-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5da674f0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5da674f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:00:02,663-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7d9d2654
>2018-05-30 01:00:02,663-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:00:02,759-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:00:02,759-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:00:02,759-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:00:02,760-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:00:02,760-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7d9d2654
>2018-05-30 01:00:02,760-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:00:02,774-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [30069ce0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:00:02,775-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [30069ce0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b5caa82
>2018-05-30 01:00:02,777-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [30069ce0] FINISH, SetVdsStatusVDSCommand, log id: b5caa82
>2018-05-30 01:00:02,808-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [30069ce0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 30069ce0, Job ID: 1e1033ab-e385-4033-80dd-e24a75f3bc81, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:00:02,813-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [30069ce0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 33a4da62, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:00:02,815-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [30069ce0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 33a4da62, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:00:02,826-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [239f7d23] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:00:02,827-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [239f7d23] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:00:03,479-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [13483ed3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4949dd81
>2018-05-30 01:00:03,481-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [13483ed3] FINISH, GetHardwareInfoVDSCommand, log id: 4949dd81
>2018-05-30 01:00:03,493-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [4eed6b56] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:00:03,502-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [7f6a7927] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:00:03,507-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [36bb71a5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:00:03,518-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [36bb71a5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 65a63f5d
>2018-05-30 01:00:03,887-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [36bb71a5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:00:03,887-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [36bb71a5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 65a63f5d
>2018-05-30 01:00:03,891-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [36bb71a5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:00:03,892-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [36bb71a5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:00:03,895-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [36bb71a5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:00:03,896-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [36bb71a5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:00:03,900-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [36bb71a5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:00:03,901-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [36bb71a5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:00:03,903-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [36bb71a5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:00:03,906-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [36bb71a5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 36bb71a5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:00:03,909-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: d30f79a
>2018-05-30 01:00:03,909-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:00:04,008-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:00:04,008-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:00:04,008-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:00:04,008-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:00:04,008-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: d30f79a
>2018-05-30 01:00:04,008-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:00:04,021-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4b136d2b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:00:04,022-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4b136d2b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41b90a48
>2018-05-30 01:00:04,024-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4b136d2b] FINISH, SetVdsStatusVDSCommand, log id: 41b90a48
>2018-05-30 01:00:04,030-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4b136d2b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4b136d2b, Job ID: 0a9abc53-d6ad-4cb0-bc4c-9c7df2fdf95d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:00:04,034-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4b136d2b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7f6a7927, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:00:04,036-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4b136d2b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7f6a7927, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:00:04,047-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [5c100f04] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:00:04,048-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [5c100f04] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7ee51af] Autorecovering 2 hosts
>2018-05-30 01:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7ee51af] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:05:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1426e774] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1426e774] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1426e774] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1426e774] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:05:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1426e774] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39295565
>2018-05-30 01:05:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1426e774] FINISH, SetVdsStatusVDSCommand, log id: 39295565
>2018-05-30 01:05:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1426e774] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:05:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1426e774] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:05:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1426e774] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:05:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [574e1661] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [574e1661] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [574e1661] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [574e1661] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:05:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [574e1661] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a09240f
>2018-05-30 01:05:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [574e1661] FINISH, SetVdsStatusVDSCommand, log id: a09240f
>2018-05-30 01:05:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [574e1661] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 118a05f7
>2018-05-30 01:05:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [574e1661] FINISH, SetHaMaintenanceModeVDSCommand, log id: 118a05f7
>2018-05-30 01:05:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [574e1661] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:05:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [574e1661] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:05:00,419-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2313b160] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2e278a51
>2018-05-30 01:05:00,421-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2313b160] FINISH, GetHardwareInfoVDSCommand, log id: 2e278a51
>2018-05-30 01:05:00,435-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6483e93d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:05:00,443-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [79720ad] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:05:00,448-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1e373c38] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:05:00,459-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1e373c38] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2605cd43
>2018-05-30 01:05:00,846-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e373c38] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:05:00,846-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1e373c38] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2605cd43
>2018-05-30 01:05:00,850-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e373c38] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:05:00,851-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e373c38] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:05:00,855-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e373c38] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:05:00,856-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e373c38] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:05:00,859-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e373c38] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:05:00,860-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e373c38] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:05:00,863-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1e373c38] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:05:00,865-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e373c38] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1e373c38, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:05:00,872-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 26ed6575
>2018-05-30 01:05:00,872-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:05:00,972-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:05:00,972-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:05:00,972-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:05:00,972-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:05:00,972-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 26ed6575
>2018-05-30 01:05:00,972-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:05:00,986-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [3f9da000] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:05:00,987-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3f9da000] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3292fd78
>2018-05-30 01:05:00,989-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3f9da000] FINISH, SetVdsStatusVDSCommand, log id: 3292fd78
>2018-05-30 01:05:01,021-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f9da000] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3f9da000, Job ID: 6983663d-e1ff-4653-877e-22a3761d9484, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:05:01,026-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f9da000] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 79720ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:05:01,028-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f9da000] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 79720ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:05:01,039-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [689e41b9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:05:01,040-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [689e41b9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:05:01,735-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [ceac5c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6bb7ea4b
>2018-05-30 01:05:01,737-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [ceac5c] FINISH, GetHardwareInfoVDSCommand, log id: 6bb7ea4b
>2018-05-30 01:05:01,750-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [33fc6e30] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:05:01,755-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [7cb72bd9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:05:01,760-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2cfc536f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:05:01,767-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2cfc536f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2dac804a
>2018-05-30 01:05:03,141-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2cfc536f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:05:03,141-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2cfc536f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2dac804a
>2018-05-30 01:05:03,145-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2cfc536f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:05:03,146-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2cfc536f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:05:03,149-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2cfc536f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:05:03,150-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2cfc536f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:05:03,154-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2cfc536f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:05:03,155-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2cfc536f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:05:03,158-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2cfc536f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:05:03,162-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2cfc536f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2cfc536f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:05:03,169-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 228cf4c4
>2018-05-30 01:05:03,169-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:05:03,265-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:05:03,265-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:05:03,265-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:05:03,265-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:05:03,265-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 228cf4c4
>2018-05-30 01:05:03,265-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:05:03,278-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [33380958] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:05:03,280-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [33380958] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 9b77a10
>2018-05-30 01:05:03,282-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [33380958] FINISH, SetVdsStatusVDSCommand, log id: 9b77a10
>2018-05-30 01:05:03,314-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [33380958] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 33380958, Job ID: 943c8276-6456-447b-9107-a72b02e7be2a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:05:03,319-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [33380958] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7cb72bd9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:05:03,321-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [33380958] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7cb72bd9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:05:03,332-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [3f6d1dfc] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:05:03,333-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [3f6d1dfc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:10:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6ea69e5e] Autorecovering 2 hosts
>2018-05-30 01:10:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6ea69e5e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:10:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d54a04e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:10:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d54a04e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:10:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d54a04e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:10:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d54a04e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:10:00,050-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6d54a04e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4c42094
>2018-05-30 01:10:00,053-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6d54a04e] FINISH, SetVdsStatusVDSCommand, log id: 4c42094
>2018-05-30 01:10:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d54a04e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:10:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d54a04e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:10:00,056-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6d54a04e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:10:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [396ed3b3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:10:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [396ed3b3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:10:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [396ed3b3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:10:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [396ed3b3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:10:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [396ed3b3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54061b0a
>2018-05-30 01:10:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [396ed3b3] FINISH, SetVdsStatusVDSCommand, log id: 54061b0a
>2018-05-30 01:10:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [396ed3b3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6905d62
>2018-05-30 01:10:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [396ed3b3] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6905d62
>2018-05-30 01:10:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [396ed3b3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:10:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [396ed3b3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:10:01,088-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [75d87e39] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 45857dc1
>2018-05-30 01:10:01,089-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [75d87e39] FINISH, GetHardwareInfoVDSCommand, log id: 45857dc1
>2018-05-30 01:10:01,102-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7c922630] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:10:01,111-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3d34bb5a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:10:01,115-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [420e0c82] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:10:01,122-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [420e0c82] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7e357c58
>2018-05-30 01:10:01,492-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [420e0c82] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:10:01,492-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [420e0c82] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7e357c58
>2018-05-30 01:10:01,497-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [420e0c82] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:10:01,497-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [420e0c82] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:10:01,501-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [420e0c82] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:10:01,502-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [420e0c82] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:10:01,506-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [420e0c82] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:10:01,506-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [420e0c82] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:10:01,510-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [420e0c82] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:10:01,512-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [420e0c82] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 420e0c82, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:10:01,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 18ebeae9
>2018-05-30 01:10:01,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:10:01,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:10:01,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:10:01,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:10:01,615-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:10:01,615-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 18ebeae9
>2018-05-30 01:10:01,615-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:10:01,629-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [1fd32e53] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:10:01,630-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1fd32e53] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2188da79
>2018-05-30 01:10:01,632-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1fd32e53] FINISH, SetVdsStatusVDSCommand, log id: 2188da79
>2018-05-30 01:10:01,662-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1fd32e53] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1fd32e53, Job ID: f9c3722b-5920-4315-9d02-6f551f3dc94a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:10:01,667-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1fd32e53] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3d34bb5a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:10:01,669-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1fd32e53] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3d34bb5a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:10:01,680-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [6c6b191b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:10:01,681-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [6c6b191b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:10:01,771-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1799f3e6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 339edba
>2018-05-30 01:10:01,772-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1799f3e6] FINISH, GetHardwareInfoVDSCommand, log id: 339edba
>2018-05-30 01:10:01,786-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [689d5056] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:10:01,790-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [24aa97af] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:10:01,795-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [507215f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:10:01,802-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [507215f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2954ae0b
>2018-05-30 01:10:02,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [507215f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:10:02,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [507215f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2954ae0b
>2018-05-30 01:10:02,182-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [507215f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:10:02,183-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [507215f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:10:02,187-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [507215f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:10:02,187-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [507215f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:10:02,191-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [507215f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:10:02,192-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [507215f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:10:02,195-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [507215f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:10:02,197-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [507215f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 507215f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:10:02,201-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 463c46c0
>2018-05-30 01:10:02,201-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:10:02,296-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:10:02,296-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:10:02,297-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:10:02,297-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:10:02,297-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 463c46c0
>2018-05-30 01:10:02,297-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:10:02,311-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [511f5dba] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:10:02,312-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [511f5dba] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47bff17d
>2018-05-30 01:10:02,314-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [511f5dba] FINISH, SetVdsStatusVDSCommand, log id: 47bff17d
>2018-05-30 01:10:02,320-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [511f5dba] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 511f5dba, Job ID: 3e874191-b447-447a-b0ba-962ffeca3c6c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:10:02,325-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [511f5dba] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 24aa97af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:10:02,327-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [511f5dba] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 24aa97af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:10:02,339-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [580767a9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:10:02,340-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [580767a9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:15:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [554b8631] Autorecovering 2 hosts
>2018-05-30 01:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [554b8631] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:15:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [582d0662] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:15:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [582d0662] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:15:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [582d0662] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:15:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [582d0662] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:15:00,047-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [582d0662] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 29111a65
>2018-05-30 01:15:00,050-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [582d0662] FINISH, SetVdsStatusVDSCommand, log id: 29111a65
>2018-05-30 01:15:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [582d0662] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:15:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [582d0662] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:15:00,053-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [582d0662] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:15:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [72d65870] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:15:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [72d65870] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:15:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [72d65870] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:15:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [72d65870] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:15:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [72d65870] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39de022
>2018-05-30 01:15:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [72d65870] FINISH, SetVdsStatusVDSCommand, log id: 39de022
>2018-05-30 01:15:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [72d65870] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 73dbc765
>2018-05-30 01:15:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [72d65870] FINISH, SetHaMaintenanceModeVDSCommand, log id: 73dbc765
>2018-05-30 01:15:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [72d65870] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:15:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [72d65870] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:15:02,361-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [f99ffae] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: a0feeed
>2018-05-30 01:15:02,363-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [f99ffae] FINISH, GetHardwareInfoVDSCommand, log id: a0feeed
>2018-05-30 01:15:02,377-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [501f21e8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:15:02,386-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [66f5999f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:15:02,391-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5050a073] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:15:02,449-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5050a073] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: b17e98d
>2018-05-30 01:15:02,903-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [79838e7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 382485da
>2018-05-30 01:15:02,904-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [79838e7] FINISH, GetHardwareInfoVDSCommand, log id: 382485da
>2018-05-30 01:15:02,919-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [48fcec1f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:15:02,928-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [322952b9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:15:02,933-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [50f4076a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:15:02,945-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [50f4076a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: b8bd964
>2018-05-30 01:15:03,319-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [50f4076a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:15:03,319-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [50f4076a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: b8bd964
>2018-05-30 01:15:03,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [50f4076a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:15:03,324-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [50f4076a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:15:03,328-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [50f4076a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:15:03,329-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [50f4076a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:15:03,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [50f4076a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:15:03,333-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [50f4076a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:15:03,336-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [50f4076a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:15:03,338-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [50f4076a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 50f4076a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:15:03,345-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5f30919a
>2018-05-30 01:15:03,345-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:15:03,443-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:15:03,443-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:15:03,443-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:15:03,443-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:15:03,443-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5f30919a
>2018-05-30 01:15:03,443-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:15:03,456-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [7c65bcc9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:15:03,457-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7c65bcc9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f4155c4
>2018-05-30 01:15:03,459-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7c65bcc9] FINISH, SetVdsStatusVDSCommand, log id: 5f4155c4
>2018-05-30 01:15:03,493-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7c65bcc9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7c65bcc9, Job ID: 3cfed5e3-d537-435e-ac56-64225f094b2d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:15:03,498-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7c65bcc9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 322952b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:15:03,500-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7c65bcc9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 322952b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:15:03,511-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [3be61973] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:15:03,512-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [3be61973] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:15:03,814-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5050a073] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:15:03,814-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5050a073] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: b17e98d
>2018-05-30 01:15:03,818-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5050a073] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:15:03,818-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5050a073] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:15:03,822-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5050a073] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:15:03,823-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5050a073] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:15:03,827-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5050a073] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:15:03,827-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5050a073] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:15:03,831-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5050a073] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:15:03,833-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5050a073] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5050a073, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:15:03,836-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 42355f91
>2018-05-30 01:15:03,836-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:15:03,933-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:15:03,933-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:15:03,933-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:15:03,933-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:15:03,933-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 42355f91
>2018-05-30 01:15:03,933-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:15:03,946-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [156d8645] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:15:03,947-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [156d8645] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 226ef1c7
>2018-05-30 01:15:03,949-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [156d8645] FINISH, SetVdsStatusVDSCommand, log id: 226ef1c7
>2018-05-30 01:15:03,955-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [156d8645] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 156d8645, Job ID: 697df65c-0aed-4033-952f-66147196081a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:15:03,959-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [156d8645] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 66f5999f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:15:03,962-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [156d8645] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 66f5999f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:15:03,972-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [60e0e2a7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:15:03,973-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [60e0e2a7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:20:00,014-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7eb5043d] Autorecovering 2 hosts
>2018-05-30 01:20:00,014-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7eb5043d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:20:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [b585c0f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:20:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [b585c0f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:20:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [b585c0f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:20:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [b585c0f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:20:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [b585c0f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f58ebb4
>2018-05-30 01:20:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [b585c0f] FINISH, SetVdsStatusVDSCommand, log id: 5f58ebb4
>2018-05-30 01:20:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [b585c0f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:20:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [b585c0f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:20:00,082-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [b585c0f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:20:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4325cb1e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:20:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4325cb1e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:20:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4325cb1e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:20:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4325cb1e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:20:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4325cb1e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a5177e
>2018-05-30 01:20:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4325cb1e] FINISH, SetVdsStatusVDSCommand, log id: a5177e
>2018-05-30 01:20:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4325cb1e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 36462e7
>2018-05-30 01:20:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4325cb1e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 36462e7
>2018-05-30 01:20:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4325cb1e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:20:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4325cb1e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:20:01,074-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4a7429c3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 363ad5f9
>2018-05-30 01:20:01,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4a7429c3] FINISH, GetHardwareInfoVDSCommand, log id: 363ad5f9
>2018-05-30 01:20:01,089-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [720dd4b5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:20:01,098-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [854a52c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:20:01,103-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4efd1a24] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:20:01,114-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4efd1a24] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2276799c
>2018-05-30 01:20:01,502-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4efd1a24] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:20:01,502-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4efd1a24] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2276799c
>2018-05-30 01:20:01,506-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4efd1a24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:20:01,507-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4efd1a24] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:20:01,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4efd1a24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:20:01,511-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4efd1a24] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:20:01,516-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4efd1a24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:20:01,516-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4efd1a24] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:20:01,520-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4efd1a24] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:20:01,523-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4efd1a24] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4efd1a24, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:20:01,529-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 15090286
>2018-05-30 01:20:01,529-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:20:01,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:20:01,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:20:01,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:20:01,627-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:20:01,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 15090286
>2018-05-30 01:20:01,627-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:20:01,639-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4650536c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: cfd933b
>2018-05-30 01:20:01,641-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4650536c] FINISH, GetHardwareInfoVDSCommand, log id: cfd933b
>2018-05-30 01:20:01,642-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [fa6f589] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:20:01,644-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [fa6f589] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 73e61b33
>2018-05-30 01:20:01,646-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [fa6f589] FINISH, SetVdsStatusVDSCommand, log id: 73e61b33
>2018-05-30 01:20:01,658-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [c873ab5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:20:01,664-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [4e1d5bf7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:20:01,671-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [169e92e6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:20:01,682-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [169e92e6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f1f8fda
>2018-05-30 01:20:01,691-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [fa6f589] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: fa6f589, Job ID: 48ad0161-3799-429c-886a-4bffaeb4e3f1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:20:01,695-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [fa6f589] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 854a52c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:20:01,698-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [fa6f589] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 854a52c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:20:01,708-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [3c109678] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:20:01,710-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [3c109678] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:20:02,055-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169e92e6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:20:02,055-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [169e92e6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f1f8fda
>2018-05-30 01:20:02,058-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169e92e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:20:02,059-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [169e92e6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:20:02,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169e92e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:20:02,064-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [169e92e6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:20:02,067-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169e92e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:20:02,068-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [169e92e6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:20:02,071-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [169e92e6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:20:02,074-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169e92e6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 169e92e6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:20:02,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 9398add
>2018-05-30 01:20:02,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:20:02,175-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:20:02,175-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:20:02,175-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:20:02,175-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:20:02,175-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 9398add
>2018-05-30 01:20:02,175-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:20:02,189-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [60fbcf72] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:20:02,190-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [60fbcf72] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25084b83
>2018-05-30 01:20:02,192-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [60fbcf72] FINISH, SetVdsStatusVDSCommand, log id: 25084b83
>2018-05-30 01:20:02,198-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [60fbcf72] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 60fbcf72, Job ID: 46b604eb-8b0b-4225-87fc-94eecc51dd37, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:20:02,203-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [60fbcf72] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4e1d5bf7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:20:02,205-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [60fbcf72] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4e1d5bf7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:20:02,216-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [4f5f1adf] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:20:02,217-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [4f5f1adf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5f8dcdef] Autorecovering 2 hosts
>2018-05-30 01:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5f8dcdef] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:25:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b99fa1b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:25:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b99fa1b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:25:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b99fa1b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:25:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b99fa1b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:25:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5b99fa1b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 23c2ce3f
>2018-05-30 01:25:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5b99fa1b] FINISH, SetVdsStatusVDSCommand, log id: 23c2ce3f
>2018-05-30 01:25:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b99fa1b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:25:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b99fa1b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:25:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5b99fa1b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:25:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2290e22d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2290e22d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2290e22d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2290e22d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:25:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2290e22d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d8c8321
>2018-05-30 01:25:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2290e22d] FINISH, SetVdsStatusVDSCommand, log id: d8c8321
>2018-05-30 01:25:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [2290e22d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6d3e4d3c
>2018-05-30 01:25:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [2290e22d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6d3e4d3c
>2018-05-30 01:25:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2290e22d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:25:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2290e22d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:25:02,298-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7fc19a1b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 20b4e7a4
>2018-05-30 01:25:02,300-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7fc19a1b] FINISH, GetHardwareInfoVDSCommand, log id: 20b4e7a4
>2018-05-30 01:25:02,314-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [4b00876d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:25:02,322-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [1896678f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:25:02,328-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2047a57b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:25:02,339-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2047a57b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 29d7fa2e
>2018-05-30 01:25:02,723-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2047a57b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:25:02,723-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2047a57b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 29d7fa2e
>2018-05-30 01:25:02,727-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2047a57b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:25:02,727-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2047a57b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:25:02,731-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2047a57b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:25:02,732-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2047a57b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:25:02,736-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2047a57b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:25:02,737-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2047a57b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:25:02,740-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2047a57b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:25:02,742-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2047a57b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2047a57b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:25:02,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 351d5e94
>2018-05-30 01:25:02,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:25:02,846-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:25:02,847-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:25:02,847-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:25:02,847-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:25:02,847-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 351d5e94
>2018-05-30 01:25:02,847-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:25:02,861-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [349a6866] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:25:02,862-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [349a6866] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6db8138f
>2018-05-30 01:25:02,864-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [349a6866] FINISH, SetVdsStatusVDSCommand, log id: 6db8138f
>2018-05-30 01:25:02,896-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [349a6866] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 349a6866, Job ID: 48e71374-1b4b-4697-be28-bd1b8bd652b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:25:02,902-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [349a6866] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1896678f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:25:02,904-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [349a6866] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1896678f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:25:02,904-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7cef114d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 20243f52
>2018-05-30 01:25:02,906-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7cef114d] FINISH, GetHardwareInfoVDSCommand, log id: 20243f52
>2018-05-30 01:25:02,916-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [3e1c1aae] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:25:02,917-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [3e1c1aae] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:25:02,918-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [76c4223c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:25:02,923-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [7df81cb6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:25:02,928-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3a38307a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:25:02,935-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3a38307a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 486e5710
>2018-05-30 01:25:03,304-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3a38307a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:25:03,304-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3a38307a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 486e5710
>2018-05-30 01:25:03,308-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3a38307a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:25:03,308-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3a38307a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:25:03,312-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3a38307a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:25:03,313-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3a38307a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:25:03,317-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3a38307a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:25:03,318-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3a38307a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:25:03,320-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3a38307a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:25:03,323-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3a38307a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3a38307a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:25:03,330-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1cf882be
>2018-05-30 01:25:03,330-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:25:03,427-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:25:03,427-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:25:03,427-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:25:03,427-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:25:03,427-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1cf882be
>2018-05-30 01:25:03,427-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:25:03,440-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [496be5c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:25:03,441-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [496be5c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bddbb25
>2018-05-30 01:25:03,443-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [496be5c] FINISH, SetVdsStatusVDSCommand, log id: 3bddbb25
>2018-05-30 01:25:03,475-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [496be5c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 496be5c, Job ID: 5d2379a2-baca-4c49-bfb6-0482b1ee88cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:25:03,480-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [496be5c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7df81cb6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:25:03,482-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [496be5c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7df81cb6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:25:03,493-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [310ad4e5] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:25:03,494-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [310ad4e5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6b6c7289] Autorecovering 2 hosts
>2018-05-30 01:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6b6c7289] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:30:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [e2431a7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:30:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [e2431a7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:30:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [e2431a7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:30:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [e2431a7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:30:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [e2431a7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41c78cb7
>2018-05-30 01:30:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [e2431a7] FINISH, SetVdsStatusVDSCommand, log id: 41c78cb7
>2018-05-30 01:30:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [e2431a7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:30:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [e2431a7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:30:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [e2431a7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:30:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d84dc5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:30:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d84dc5] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:30:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d84dc5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:30:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d84dc5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:30:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3d84dc5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d7e47e3
>2018-05-30 01:30:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3d84dc5] FINISH, SetVdsStatusVDSCommand, log id: 3d7e47e3
>2018-05-30 01:30:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [3d84dc5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7cac73f0
>2018-05-30 01:30:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [3d84dc5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7cac73f0
>2018-05-30 01:30:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d84dc5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:30:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d84dc5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:30:00,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [5e3dff16] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 28951bf0
>2018-05-30 01:30:00,524-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [5e3dff16] FINISH, GetHardwareInfoVDSCommand, log id: 28951bf0
>2018-05-30 01:30:00,536-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [37163a23] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:30:00,545-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [2306491e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:30:00,550-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [27427e8d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:30:00,561-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [27427e8d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4bb7243a
>2018-05-30 01:30:00,941-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [27427e8d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:30:00,941-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [27427e8d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4bb7243a
>2018-05-30 01:30:00,946-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [27427e8d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:30:00,946-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [27427e8d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:30:00,950-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [27427e8d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:30:00,951-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [27427e8d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:30:00,955-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [27427e8d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:30:00,955-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [27427e8d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:30:00,958-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [27427e8d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:30:00,961-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [27427e8d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 27427e8d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:30:00,967-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5b60c8d0
>2018-05-30 01:30:00,967-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:30:01,062-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:30:01,062-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:30:01,062-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:30:01,062-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:30:01,062-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5b60c8d0
>2018-05-30 01:30:01,062-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:30:01,076-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [77c9a963] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:30:01,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [77c9a963] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 27aca4ed
>2018-05-30 01:30:01,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [77c9a963] FINISH, SetVdsStatusVDSCommand, log id: 27aca4ed
>2018-05-30 01:30:01,088-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [77c9a963] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 77c9a963, Job ID: b7eec72f-9f17-461f-820c-fecb9b91496c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:30:01,093-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [77c9a963] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2306491e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:30:01,095-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [77c9a963] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2306491e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:30:01,106-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7f5c58f0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:30:01,107-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7f5c58f0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:30:01,151-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [57d24394] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 691a40d4
>2018-05-30 01:30:01,152-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [57d24394] FINISH, GetHardwareInfoVDSCommand, log id: 691a40d4
>2018-05-30 01:30:01,165-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7646d366] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:30:01,170-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [29d14073] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:30:01,175-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [34cbf2af] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:30:01,238-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [34cbf2af] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 516316a7
>2018-05-30 01:30:01,617-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [34cbf2af] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:30:01,617-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [34cbf2af] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 516316a7
>2018-05-30 01:30:01,621-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [34cbf2af] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:30:01,622-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [34cbf2af] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:30:01,626-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [34cbf2af] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:30:01,626-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [34cbf2af] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:30:01,630-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [34cbf2af] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:30:01,631-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [34cbf2af] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:30:01,634-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [34cbf2af] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:30:01,636-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [34cbf2af] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 34cbf2af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:30:01,642-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 36befc2b
>2018-05-30 01:30:01,642-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:30:01,740-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:30:01,740-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:30:01,740-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:30:01,740-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:30:01,740-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 36befc2b
>2018-05-30 01:30:01,740-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:30:01,753-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [21eb1cb4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:30:01,755-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [21eb1cb4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 243fa512
>2018-05-30 01:30:01,757-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [21eb1cb4] FINISH, SetVdsStatusVDSCommand, log id: 243fa512
>2018-05-30 01:30:01,763-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21eb1cb4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 21eb1cb4, Job ID: 495deb3e-0124-49c0-befb-2b3887bf6882, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:30:01,767-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21eb1cb4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29d14073, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:30:01,769-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21eb1cb4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29d14073, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:30:01,780-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [3056928a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:30:01,781-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [3056928a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:35:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [67b8bb0e] Autorecovering 2 hosts
>2018-05-30 01:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [67b8bb0e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:35:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3ed531bb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:35:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3ed531bb] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:35:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3ed531bb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:35:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3ed531bb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:35:00,050-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3ed531bb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1081d075
>2018-05-30 01:35:00,052-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3ed531bb] FINISH, SetVdsStatusVDSCommand, log id: 1081d075
>2018-05-30 01:35:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3ed531bb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:35:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3ed531bb] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:35:00,055-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [3ed531bb] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:35:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [54dd20c0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:35:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [54dd20c0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:35:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [54dd20c0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:35:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [54dd20c0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:35:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [54dd20c0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45cbafea
>2018-05-30 01:35:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [54dd20c0] FINISH, SetVdsStatusVDSCommand, log id: 45cbafea
>2018-05-30 01:35:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [54dd20c0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 56e10e03
>2018-05-30 01:35:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [54dd20c0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 56e10e03
>2018-05-30 01:35:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [54dd20c0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:35:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [54dd20c0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:35:01,743-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [43c2101d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 64c5d1b0
>2018-05-30 01:35:01,745-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [43c2101d] FINISH, GetHardwareInfoVDSCommand, log id: 64c5d1b0
>2018-05-30 01:35:01,759-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [392df5b9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:35:01,768-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [21586d13] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:35:01,772-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6a9319ed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:35:01,779-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6a9319ed] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4634801b
>2018-05-30 01:35:02,154-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a9319ed] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:35:02,154-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6a9319ed] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4634801b
>2018-05-30 01:35:02,158-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a9319ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:35:02,158-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6a9319ed] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:35:02,162-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a9319ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:35:02,163-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6a9319ed] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:35:02,166-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a9319ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:35:02,167-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6a9319ed] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:35:02,170-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6a9319ed] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:35:02,173-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a9319ed] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6a9319ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:35:02,176-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1299ae5f
>2018-05-30 01:35:02,176-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:35:02,274-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:35:02,274-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:35:02,274-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:35:02,274-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:35:02,274-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1299ae5f
>2018-05-30 01:35:02,274-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:35:02,287-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [404ad4f5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:35:02,288-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [404ad4f5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12351a1d
>2018-05-30 01:35:02,290-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [404ad4f5] FINISH, SetVdsStatusVDSCommand, log id: 12351a1d
>2018-05-30 01:35:02,321-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [404ad4f5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 404ad4f5, Job ID: 75301892-3963-4bd2-a87d-146f80e99420, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:35:02,326-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [404ad4f5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 21586d13, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:35:02,329-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [404ad4f5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 21586d13, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:35:02,340-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [77d9a378] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:35:02,341-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [77d9a378] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:35:02,533-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7aa2a974] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1a9b4b1
>2018-05-30 01:35:02,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7aa2a974] FINISH, GetHardwareInfoVDSCommand, log id: 1a9b4b1
>2018-05-30 01:35:02,548-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [301aee4b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:35:02,552-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [401dd8f9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:35:02,557-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5c87be8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:35:02,564-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5c87be8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7cbbae8
>2018-05-30 01:35:03,931-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5c87be8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:35:03,931-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5c87be8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7cbbae8
>2018-05-30 01:35:03,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5c87be8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:35:03,935-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5c87be8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:35:03,939-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5c87be8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:35:03,940-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5c87be8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:35:03,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5c87be8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:35:03,945-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5c87be8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:35:03,948-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5c87be8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:35:03,950-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5c87be8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5c87be8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:35:03,953-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 743c6b76
>2018-05-30 01:35:03,953-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:35:04,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:35:04,052-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:35:04,052-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:35:04,052-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:35:04,052-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 743c6b76
>2018-05-30 01:35:04,052-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:35:04,065-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [72a068e5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:35:04,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [72a068e5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 68f17b06
>2018-05-30 01:35:04,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [72a068e5] FINISH, SetVdsStatusVDSCommand, log id: 68f17b06
>2018-05-30 01:35:04,075-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [72a068e5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 72a068e5, Job ID: e9eba01a-082d-42ba-ab10-9d2186d0e72d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:35:04,079-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [72a068e5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 401dd8f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:35:04,082-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [72a068e5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 401dd8f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:35:04,093-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5a03d551] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:35:04,094-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5a03d551] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [49554884] Autorecovering 2 hosts
>2018-05-30 01:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [49554884] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:40:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [110a6435] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:40:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [110a6435] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:40:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [110a6435] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:40:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [110a6435] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:40:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [110a6435] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 561d3177
>2018-05-30 01:40:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [110a6435] FINISH, SetVdsStatusVDSCommand, log id: 561d3177
>2018-05-30 01:40:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [110a6435] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:40:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [110a6435] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:40:00,082-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [110a6435] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:40:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5c97e39] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:40:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5c97e39] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:40:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5c97e39] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:40:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5c97e39] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:40:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5c97e39] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3c6cc06c
>2018-05-30 01:40:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5c97e39] FINISH, SetVdsStatusVDSCommand, log id: 3c6cc06c
>2018-05-30 01:40:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5c97e39] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 57033fa8
>2018-05-30 01:40:00,103-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5c97e39] FINISH, SetHaMaintenanceModeVDSCommand, log id: 57033fa8
>2018-05-30 01:40:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5c97e39] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:40:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5c97e39] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:40:01,777-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1e6c3423] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 11777fe6
>2018-05-30 01:40:01,779-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1e6c3423] FINISH, GetHardwareInfoVDSCommand, log id: 11777fe6
>2018-05-30 01:40:01,792-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [7298f72b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:40:01,801-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [11f3449d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:40:01,806-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6aa07de1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:40:01,812-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6aa07de1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 105dc49
>2018-05-30 01:40:02,184-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6aa07de1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:40:02,184-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6aa07de1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 105dc49
>2018-05-30 01:40:02,188-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6aa07de1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:40:02,189-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6aa07de1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:40:02,193-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6aa07de1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:40:02,193-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6aa07de1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:40:02,197-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6aa07de1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:40:02,198-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6aa07de1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:40:02,201-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6aa07de1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:40:02,204-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6aa07de1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6aa07de1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:40:02,207-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 25a7220d
>2018-05-30 01:40:02,207-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:40:02,305-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:40:02,305-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:40:02,305-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:40:02,305-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:40:02,305-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 25a7220d
>2018-05-30 01:40:02,305-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:40:02,319-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2aa62886] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:40:02,320-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2aa62886] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3dbf24a5
>2018-05-30 01:40:02,322-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2aa62886] FINISH, SetVdsStatusVDSCommand, log id: 3dbf24a5
>2018-05-30 01:40:02,353-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2aa62886] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2aa62886, Job ID: 1bb9332c-4c5e-466f-87ad-f877d955f3ee, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:40:02,357-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2aa62886] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 11f3449d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:40:02,360-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2aa62886] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 11f3449d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:40:02,371-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [24ed849a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:40:02,372-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [24ed849a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:40:02,975-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [615f82db] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 43f45ab2
>2018-05-30 01:40:02,977-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [615f82db] FINISH, GetHardwareInfoVDSCommand, log id: 43f45ab2
>2018-05-30 01:40:02,991-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [5eb5c05c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:40:02,995-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [6b079332] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:40:03,001-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [165b596] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:40:03,011-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [165b596] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 789ba53
>2018-05-30 01:40:03,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [165b596] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:40:03,388-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [165b596] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 789ba53
>2018-05-30 01:40:03,392-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [165b596] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:40:03,393-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [165b596] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:40:03,397-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [165b596] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:40:03,398-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [165b596] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:40:03,402-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [165b596] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:40:03,402-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [165b596] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:40:03,406-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [165b596] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:40:03,409-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [165b596] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 165b596, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:40:03,412-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 550a346e
>2018-05-30 01:40:03,412-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:40:03,510-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:40:03,510-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:40:03,510-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:40:03,510-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:40:03,510-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 550a346e
>2018-05-30 01:40:03,510-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:40:03,523-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [43218d6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:40:03,525-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [43218d6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 497e7a57
>2018-05-30 01:40:03,527-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [43218d6] FINISH, SetVdsStatusVDSCommand, log id: 497e7a57
>2018-05-30 01:40:03,532-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [43218d6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 43218d6, Job ID: c7708b28-ee35-40ac-a2f9-821d81a9bf68, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:40:03,537-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [43218d6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6b079332, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:40:03,539-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [43218d6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6b079332, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:40:03,550-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [7190c127] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:40:03,551-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7190c127] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [196c4617] Autorecovering 2 hosts
>2018-05-30 01:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [196c4617] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:45:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [76911a2f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [76911a2f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [76911a2f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:45:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [76911a2f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:45:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [76911a2f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 35981d44
>2018-05-30 01:45:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [76911a2f] FINISH, SetVdsStatusVDSCommand, log id: 35981d44
>2018-05-30 01:45:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [76911a2f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:45:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [76911a2f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:45:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [76911a2f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:45:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [608c7315] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:45:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [608c7315] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:45:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [608c7315] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:45:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [608c7315] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:45:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [608c7315] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 792d426
>2018-05-30 01:45:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [608c7315] FINISH, SetVdsStatusVDSCommand, log id: 792d426
>2018-05-30 01:45:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [608c7315] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7bd95008
>2018-05-30 01:45:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [608c7315] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7bd95008
>2018-05-30 01:45:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [608c7315] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:45:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [608c7315] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:45:01,240-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4f0beda2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 359dd901
>2018-05-30 01:45:01,241-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4f0beda2] FINISH, GetHardwareInfoVDSCommand, log id: 359dd901
>2018-05-30 01:45:01,254-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4c14d7e0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:45:01,263-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [c4a66e3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:45:01,268-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3d8d6ce1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:45:01,279-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3d8d6ce1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 610bcf21
>2018-05-30 01:45:01,665-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3d8d6ce1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:45:01,665-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3d8d6ce1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 610bcf21
>2018-05-30 01:45:01,670-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3d8d6ce1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:45:01,670-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3d8d6ce1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:45:01,674-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3d8d6ce1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:45:01,675-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3d8d6ce1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:45:01,679-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3d8d6ce1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:45:01,680-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3d8d6ce1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:45:01,683-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3d8d6ce1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:45:01,685-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3d8d6ce1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3d8d6ce1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:45:01,693-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5978aba0
>2018-05-30 01:45:01,693-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:45:01,789-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:45:01,789-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:45:01,789-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:45:01,789-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:45:01,789-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5978aba0
>2018-05-30 01:45:01,790-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:45:01,803-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [3678310e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:45:01,804-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3678310e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4cba6366
>2018-05-30 01:45:01,806-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3678310e] FINISH, SetVdsStatusVDSCommand, log id: 4cba6366
>2018-05-30 01:45:01,839-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3678310e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3678310e, Job ID: 1202d194-2297-4d00-a344-e27f605a883c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:45:01,844-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3678310e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: c4a66e3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:45:01,846-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3678310e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: c4a66e3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:45:01,857-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [f1e7622] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:45:01,858-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [f1e7622] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:45:03,191-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [16248d19] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4d24d770
>2018-05-30 01:45:03,193-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [16248d19] FINISH, GetHardwareInfoVDSCommand, log id: 4d24d770
>2018-05-30 01:45:03,207-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [537ae5a3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:45:03,216-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [626b230e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:45:03,221-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [414da02a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:45:03,283-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [414da02a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 15a68701
>2018-05-30 01:45:03,651-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [414da02a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:45:03,651-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [414da02a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 15a68701
>2018-05-30 01:45:03,655-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [414da02a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:45:03,656-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [414da02a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:45:03,660-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [414da02a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:45:03,660-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [414da02a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:45:03,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [414da02a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:45:03,665-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [414da02a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:45:03,668-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [414da02a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:45:03,670-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [414da02a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 414da02a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:45:03,677-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1a97ab5b
>2018-05-30 01:45:03,677-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:45:03,775-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:45:03,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:45:03,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:45:03,775-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:45:03,775-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1a97ab5b
>2018-05-30 01:45:03,775-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:45:03,789-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [18ba37db] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:45:03,790-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [18ba37db] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b7f42d5
>2018-05-30 01:45:03,792-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [18ba37db] FINISH, SetVdsStatusVDSCommand, log id: 4b7f42d5
>2018-05-30 01:45:03,825-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18ba37db] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 18ba37db, Job ID: 78932972-784a-467d-9516-5719c1258c66, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:45:03,829-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18ba37db] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 626b230e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:45:03,832-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18ba37db] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 626b230e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:45:03,842-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [c4e8613] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:45:03,843-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [c4e8613] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [44fb3106] Autorecovering 2 hosts
>2018-05-30 01:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [44fb3106] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:50:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1f1c9e7a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:50:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1f1c9e7a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:50:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1f1c9e7a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:50:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1f1c9e7a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:50:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1f1c9e7a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c314a22
>2018-05-30 01:50:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1f1c9e7a] FINISH, SetVdsStatusVDSCommand, log id: 7c314a22
>2018-05-30 01:50:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1f1c9e7a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:50:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1f1c9e7a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:50:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [1f1c9e7a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:50:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e2fa1f7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e2fa1f7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e2fa1f7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e2fa1f7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:50:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5e2fa1f7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b981aff
>2018-05-30 01:50:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5e2fa1f7] FINISH, SetVdsStatusVDSCommand, log id: 3b981aff
>2018-05-30 01:50:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5e2fa1f7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6e290499
>2018-05-30 01:50:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5e2fa1f7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6e290499
>2018-05-30 01:50:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e2fa1f7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:50:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e2fa1f7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:50:01,536-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [74befe10] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2d3b6e8d
>2018-05-30 01:50:01,538-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [74befe10] FINISH, GetHardwareInfoVDSCommand, log id: 2d3b6e8d
>2018-05-30 01:50:01,551-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [2b341f77] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:50:01,555-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [2d2cfb05] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:50:01,560-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [36e91a48] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:50:01,566-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [36e91a48] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3e43664b
>2018-05-30 01:50:01,936-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [36e91a48] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:50:01,937-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [36e91a48] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3e43664b
>2018-05-30 01:50:01,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [36e91a48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:50:01,941-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [36e91a48] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:50:01,945-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [36e91a48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:50:01,945-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [36e91a48] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:50:01,949-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [36e91a48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:50:01,950-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [36e91a48] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:50:01,953-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [36e91a48] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:50:01,956-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [36e91a48] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 36e91a48, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:50:01,959-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 681c14f7
>2018-05-30 01:50:01,959-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:50:02,058-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:50:02,058-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:50:02,058-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:50:02,058-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:50:02,058-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 681c14f7
>2018-05-30 01:50:02,058-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:50:02,071-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [2c730c95] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:50:02,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2c730c95] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32c5dbc1
>2018-05-30 01:50:02,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2c730c95] FINISH, SetVdsStatusVDSCommand, log id: 32c5dbc1
>2018-05-30 01:50:02,081-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2c730c95] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2c730c95, Job ID: db61b7b9-80ca-4d5e-b0ed-00973473394b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:50:02,085-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2c730c95] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2d2cfb05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:50:02,088-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2c730c95] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2d2cfb05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:50:02,098-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5df8b247] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:50:02,099-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5df8b247] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:50:02,454-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [7fe7aad7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4500993e
>2018-05-30 01:50:02,456-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [7fe7aad7] FINISH, GetHardwareInfoVDSCommand, log id: 4500993e
>2018-05-30 01:50:02,469-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [5d3fe84] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:50:02,474-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [671949b3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:50:02,479-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2778b2d3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:50:02,486-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2778b2d3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4d3b9210
>2018-05-30 01:50:02,864-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2778b2d3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:50:02,864-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2778b2d3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4d3b9210
>2018-05-30 01:50:02,868-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2778b2d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:50:02,869-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2778b2d3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:50:02,872-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2778b2d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:50:02,873-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2778b2d3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:50:02,877-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2778b2d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:50:02,878-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2778b2d3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:50:02,881-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2778b2d3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:50:02,883-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2778b2d3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2778b2d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:50:02,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 35b425a2
>2018-05-30 01:50:02,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:50:02,984-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:50:02,984-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:50:02,984-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:50:02,984-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:50:02,984-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 35b425a2
>2018-05-30 01:50:02,985-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:50:02,997-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [413d5526] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:50:02,999-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [413d5526] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1a02828e
>2018-05-30 01:50:03,001-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [413d5526] FINISH, SetVdsStatusVDSCommand, log id: 1a02828e
>2018-05-30 01:50:03,006-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [413d5526] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 413d5526, Job ID: dd7e9b35-4dc3-4645-aed6-742301a43c30, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:50:03,011-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [413d5526] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 671949b3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:50:03,013-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [413d5526] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 671949b3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:50:03,024-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [2709f439] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:50:03,025-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [2709f439] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:54:38,374-04 INFO  [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler7) [413a85b8] Backup check started.
>2018-05-30 01:54:38,387-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [413a85b8] EVENT_ID: ENGINE_NO_FULL_BACKUP(9,022), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: There is no full backup available, please run engine-backup to prevent data loss in case of corruption.
>2018-05-30 01:54:38,387-04 INFO  [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler7) [413a85b8] Backup check completed.
>2018-05-30 01:54:43,461-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler6) [7cc97a42] Attempting to update VMs/Templates Ovf.
>2018-05-30 01:54:43,464-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 01:54:43,464-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 01:54:43,464-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:54:43,468-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-30 01:54:43,471-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-30 01:54:43,471-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Attempting to update template OVFs in Data Center 'Default'
>2018-05-30 01:54:43,471-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-30 01:54:43,471-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 01:54:43,472-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 01:54:43,472-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 01:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [4b6bbb16] Autorecovering 2 hosts
>2018-05-30 01:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [4b6bbb16] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:55:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [31d2ea05] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:55:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [31d2ea05] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:55:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [31d2ea05] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:55:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [31d2ea05] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:55:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [31d2ea05] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f99b734
>2018-05-30 01:55:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [31d2ea05] FINISH, SetVdsStatusVDSCommand, log id: f99b734
>2018-05-30 01:55:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [31d2ea05] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:55:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [31d2ea05] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 01:55:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [31d2ea05] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:55:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [83b03b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:55:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [83b03b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 01:55:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [83b03b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:55:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [83b03b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:55:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [83b03b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5abac7a6
>2018-05-30 01:55:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [83b03b] FINISH, SetVdsStatusVDSCommand, log id: 5abac7a6
>2018-05-30 01:55:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [83b03b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 52533a67
>2018-05-30 01:55:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [83b03b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 52533a67
>2018-05-30 01:55:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [83b03b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 01:55:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [83b03b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 01:55:00,591-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [18f5cdc3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 34cd95bf
>2018-05-30 01:55:00,592-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [18f5cdc3] FINISH, GetHardwareInfoVDSCommand, log id: 34cd95bf
>2018-05-30 01:55:00,605-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [707551ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:55:00,614-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6691ea3a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:55:00,619-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [673912d4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:55:00,672-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [673912d4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 68388e09
>2018-05-30 01:55:01,050-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [673912d4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:55:01,050-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [673912d4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 68388e09
>2018-05-30 01:55:01,054-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [673912d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:55:01,055-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [673912d4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:55:01,059-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [673912d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:55:01,060-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [673912d4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:55:01,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [673912d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:55:01,064-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [673912d4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:55:01,067-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [673912d4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:55:01,069-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [673912d4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 673912d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:55:01,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 51a1a831
>2018-05-30 01:55:01,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:55:01,173-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:55:01,173-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:55:01,173-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:55:01,173-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:55:01,173-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 51a1a831
>2018-05-30 01:55:01,173-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:55:01,187-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [7a657a1b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:55:01,189-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7a657a1b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3dca34bf
>2018-05-30 01:55:01,191-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7a657a1b] FINISH, SetVdsStatusVDSCommand, log id: 3dca34bf
>2018-05-30 01:55:01,222-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7a657a1b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7a657a1b, Job ID: 292c5266-fed5-44e7-8780-c7e8ca33259e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:55:01,227-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7a657a1b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6691ea3a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:55:01,229-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7a657a1b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6691ea3a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:55:01,240-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [4f12e790] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 01:55:01,241-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [4f12e790] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 01:55:02,956-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2709c7c1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2ff78df3
>2018-05-30 01:55:02,958-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2709c7c1] FINISH, GetHardwareInfoVDSCommand, log id: 2ff78df3
>2018-05-30 01:55:02,971-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [7c211f53] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:55:02,975-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5797db61] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:55:02,980-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [53eb27f2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 01:55:02,987-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [53eb27f2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 55e9c6f0
>2018-05-30 01:55:04,308-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [53eb27f2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 01:55:04,308-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [53eb27f2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 55e9c6f0
>2018-05-30 01:55:04,312-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [53eb27f2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:55:04,313-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [53eb27f2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:55:04,317-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [53eb27f2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:55:04,317-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [53eb27f2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:55:04,321-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [53eb27f2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 01:55:04,322-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [53eb27f2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 01:55:04,325-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [53eb27f2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 01:55:04,328-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [53eb27f2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 53eb27f2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 01:55:04,334-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 77f92c1b
>2018-05-30 01:55:04,334-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 01:55:04,433-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:55:04,433-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 01:55:04,433-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 01:55:04,433-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 01:55:04,433-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 77f92c1b
>2018-05-30 01:55:04,433-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 01:55:04,446-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [650af7c7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:55:04,447-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [650af7c7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 748593ce
>2018-05-30 01:55:04,449-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [650af7c7] FINISH, SetVdsStatusVDSCommand, log id: 748593ce
>2018-05-30 01:55:04,457-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [650af7c7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 650af7c7, Job ID: 9bab30b8-c806-49c9-82e3-c0bd44f2e35a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 01:55:04,462-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [650af7c7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5797db61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 01:55:04,464-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [650af7c7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5797db61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 01:55:04,475-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6ba333c6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 01:55:04,476-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6ba333c6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [70318fcf] Autorecovering 2 hosts
>2018-05-30 02:00:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [70318fcf] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:00:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d725ceb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:00:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d725ceb] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:00:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d725ceb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:00:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d725ceb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:00:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [d725ceb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 717e7d06
>2018-05-30 02:00:00,080-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [d725ceb] FINISH, SetVdsStatusVDSCommand, log id: 717e7d06
>2018-05-30 02:00:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d725ceb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:00:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d725ceb] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:00:00,083-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [d725ceb] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:00:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4496989f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4496989f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4496989f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:00:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4496989f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:00:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4496989f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45a634df
>2018-05-30 02:00:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4496989f] FINISH, SetVdsStatusVDSCommand, log id: 45a634df
>2018-05-30 02:00:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [4496989f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 62430700
>2018-05-30 02:00:00,104-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [4496989f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 62430700
>2018-05-30 02:00:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4496989f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:00:00,105-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4496989f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:00:01,863-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4f7cabaa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 447822e7
>2018-05-30 02:00:01,865-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4f7cabaa] FINISH, GetHardwareInfoVDSCommand, log id: 447822e7
>2018-05-30 02:00:01,877-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [2f37cdf7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:00:01,885-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [10ef15ab] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:00:01,891-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [6f60e13e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:00:01,902-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [6f60e13e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 24c8d114
>2018-05-30 02:00:02,119-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7faaa895] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1c1ce797
>2018-05-30 02:00:02,121-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7faaa895] FINISH, GetHardwareInfoVDSCommand, log id: 1c1ce797
>2018-05-30 02:00:02,133-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [6870dfff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:00:02,138-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1fe0be15] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:00:02,143-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7ae17ef0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:00:02,149-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7ae17ef0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5c374cd8
>2018-05-30 02:00:02,278-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6f60e13e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:00:02,278-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [6f60e13e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 24c8d114
>2018-05-30 02:00:02,282-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6f60e13e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:00:02,283-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [6f60e13e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:00:02,289-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6f60e13e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:00:02,290-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [6f60e13e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:00:02,294-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6f60e13e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:00:02,295-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [6f60e13e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:00:02,298-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [6f60e13e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:00:02,300-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6f60e13e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6f60e13e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:00:02,307-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4efaeabc
>2018-05-30 02:00:02,307-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:00:02,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:00:02,401-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:00:02,401-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:00:02,401-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:00:02,401-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4efaeabc
>2018-05-30 02:00:02,401-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:00:02,414-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [612850d7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:00:02,415-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [612850d7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ca6502a
>2018-05-30 02:00:02,417-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [612850d7] FINISH, SetVdsStatusVDSCommand, log id: 3ca6502a
>2018-05-30 02:00:02,450-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [612850d7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 612850d7, Job ID: 29fe176a-6cb1-4987-a507-3fd91e66a60f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:00:02,455-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [612850d7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 10ef15ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:00:02,457-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [612850d7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 10ef15ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:00:02,467-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [2ec495e3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:00:02,468-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2ec495e3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:00:02,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ae17ef0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:00:02,520-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7ae17ef0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5c374cd8
>2018-05-30 02:00:02,524-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ae17ef0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:00:02,524-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7ae17ef0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:00:02,528-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ae17ef0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:00:02,529-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7ae17ef0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:00:02,533-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ae17ef0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:00:02,534-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7ae17ef0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:00:02,537-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7ae17ef0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:00:02,539-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ae17ef0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7ae17ef0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:00:02,543-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 67b4f957
>2018-05-30 02:00:02,543-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:00:02,640-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:00:02,640-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:00:02,640-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:00:02,640-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:00:02,640-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 67b4f957
>2018-05-30 02:00:02,640-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:00:02,654-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [50cff606] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:00:02,655-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [50cff606] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ca77a27
>2018-05-30 02:00:02,657-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [50cff606] FINISH, SetVdsStatusVDSCommand, log id: 6ca77a27
>2018-05-30 02:00:02,663-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [50cff606] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 50cff606, Job ID: e634146d-de21-4c0e-941e-5a58b79616d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:00:02,667-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [50cff606] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1fe0be15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:00:02,669-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [50cff606] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1fe0be15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:00:02,681-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [5b62ad08] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:00:02,682-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [5b62ad08] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [553c0213] Autorecovering 2 hosts
>2018-05-30 02:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [553c0213] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:05:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [63e97862] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [63e97862] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [63e97862] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [63e97862] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:05:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [63e97862] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1fd7f5cd
>2018-05-30 02:05:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [63e97862] FINISH, SetVdsStatusVDSCommand, log id: 1fd7f5cd
>2018-05-30 02:05:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [63e97862] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:05:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [63e97862] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:05:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [63e97862] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:05:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1c57127d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1c57127d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1c57127d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1c57127d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:05:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1c57127d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53c2b105
>2018-05-30 02:05:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1c57127d] FINISH, SetVdsStatusVDSCommand, log id: 53c2b105
>2018-05-30 02:05:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [1c57127d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 42938716
>2018-05-30 02:05:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [1c57127d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 42938716
>2018-05-30 02:05:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1c57127d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:05:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1c57127d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:05:00,324-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [103cbc57] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7752a995
>2018-05-30 02:05:00,326-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [103cbc57] FINISH, GetHardwareInfoVDSCommand, log id: 7752a995
>2018-05-30 02:05:00,339-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [fcc8b3a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:05:00,343-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [15bc10ec] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:05:00,348-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3265e880] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:05:00,356-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3265e880] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 74a0bc3
>2018-05-30 02:05:00,726-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3265e880] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:05:00,726-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3265e880] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 74a0bc3
>2018-05-30 02:05:00,730-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3265e880] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:05:00,731-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3265e880] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:05:00,735-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3265e880] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:05:00,736-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3265e880] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:05:00,739-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3265e880] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:05:00,740-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3265e880] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:05:00,743-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3265e880] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:05:00,745-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3265e880] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3265e880, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:05:00,748-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1d2bed53
>2018-05-30 02:05:00,748-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:05:00,846-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:05:00,846-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:05:00,846-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:05:00,846-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:05:00,846-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1d2bed53
>2018-05-30 02:05:00,846-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:05:00,859-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [2ae27700] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:05:00,860-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2ae27700] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33035567
>2018-05-30 02:05:00,863-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2ae27700] FINISH, SetVdsStatusVDSCommand, log id: 33035567
>2018-05-30 02:05:00,868-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ae27700] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2ae27700, Job ID: 6dfbca09-f0c4-4659-aa67-a07ebe2a559b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:05:00,873-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ae27700] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 15bc10ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:05:00,875-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ae27700] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 15bc10ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:05:00,886-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [31def34f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:05:00,887-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [31def34f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:05:03,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6f97d848] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 62538582
>2018-05-30 02:05:03,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6f97d848] FINISH, GetHardwareInfoVDSCommand, log id: 62538582
>2018-05-30 02:05:03,107-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [2453922c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:05:03,111-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [2417b762] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:05:03,116-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3a3b63a9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:05:03,123-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3a3b63a9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 16ea0cb1
>2018-05-30 02:05:03,504-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a3b63a9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:05:03,504-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3a3b63a9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 16ea0cb1
>2018-05-30 02:05:03,509-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a3b63a9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:05:03,509-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3a3b63a9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:05:03,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a3b63a9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:05:03,514-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3a3b63a9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:05:03,518-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a3b63a9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:05:03,519-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3a3b63a9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:05:03,522-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3a3b63a9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:05:03,524-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a3b63a9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3a3b63a9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:05:03,527-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 399edc4f
>2018-05-30 02:05:03,527-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:05:03,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:05:03,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:05:03,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:05:03,627-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:05:03,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 399edc4f
>2018-05-30 02:05:03,627-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:05:03,640-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [6a17604a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:05:03,641-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6a17604a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 725ac20a
>2018-05-30 02:05:03,643-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6a17604a] FINISH, SetVdsStatusVDSCommand, log id: 725ac20a
>2018-05-30 02:05:03,649-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a17604a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6a17604a, Job ID: 4b70c373-82ef-40f0-9605-ba5314191648, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:05:03,653-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a17604a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2417b762, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:05:03,655-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a17604a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2417b762, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:05:03,666-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [410def0d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:05:03,667-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [410def0d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [42939036] Autorecovering 2 hosts
>2018-05-30 02:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [42939036] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:10:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21a4ec32] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21a4ec32] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21a4ec32] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21a4ec32] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:10:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [21a4ec32] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6f86c28a
>2018-05-30 02:10:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [21a4ec32] FINISH, SetVdsStatusVDSCommand, log id: 6f86c28a
>2018-05-30 02:10:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21a4ec32] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:10:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21a4ec32] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:10:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [21a4ec32] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:10:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7c8bfc8c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7c8bfc8c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7c8bfc8c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:10:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7c8bfc8c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:10:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7c8bfc8c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ab960ed
>2018-05-30 02:10:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7c8bfc8c] FINISH, SetVdsStatusVDSCommand, log id: 2ab960ed
>2018-05-30 02:10:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [7c8bfc8c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 12c1a927
>2018-05-30 02:10:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [7c8bfc8c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 12c1a927
>2018-05-30 02:10:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7c8bfc8c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:10:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7c8bfc8c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:10:01,261-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [56cdd912] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 37bd412e
>2018-05-30 02:10:01,263-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [56cdd912] FINISH, GetHardwareInfoVDSCommand, log id: 37bd412e
>2018-05-30 02:10:01,277-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7fe9adc8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:10:01,285-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [64cf5b00] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:10:01,290-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [1d86b6e4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:10:01,301-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [1d86b6e4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 48dc7ea2
>2018-05-30 02:10:01,495-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [728aaea3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 19ceb896
>2018-05-30 02:10:01,497-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [728aaea3] FINISH, GetHardwareInfoVDSCommand, log id: 19ceb896
>2018-05-30 02:10:01,510-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4e4eaf28] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:10:01,514-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [384cba1d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:10:01,519-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6620d3b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:10:01,525-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6620d3b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 25ecd7ec
>2018-05-30 02:10:01,676-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1d86b6e4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:10:01,676-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [1d86b6e4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 48dc7ea2
>2018-05-30 02:10:01,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1d86b6e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:10:01,681-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [1d86b6e4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:10:01,684-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1d86b6e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:10:01,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [1d86b6e4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:10:01,689-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1d86b6e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:10:01,690-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [1d86b6e4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:10:01,692-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [1d86b6e4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:10:01,695-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1d86b6e4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1d86b6e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:10:01,701-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2555b5
>2018-05-30 02:10:01,702-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:10:01,798-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:10:01,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:10:01,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:10:01,798-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:10:01,798-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2555b5
>2018-05-30 02:10:01,798-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:10:01,811-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [43add784] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:10:01,812-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [43add784] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 684b596a
>2018-05-30 02:10:01,814-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [43add784] FINISH, SetVdsStatusVDSCommand, log id: 684b596a
>2018-05-30 02:10:01,846-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [43add784] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 43add784, Job ID: 43c8e7e2-ca2e-424a-9fc9-4a4700c36b63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:10:01,851-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [43add784] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 64cf5b00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:10:01,853-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [43add784] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 64cf5b00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:10:01,864-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [623434c6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:10:01,865-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [623434c6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:10:01,892-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6620d3b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:10:01,892-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6620d3b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 25ecd7ec
>2018-05-30 02:10:01,895-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6620d3b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:10:01,896-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6620d3b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:10:01,900-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6620d3b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:10:01,901-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6620d3b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:10:01,905-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6620d3b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:10:01,906-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6620d3b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:10:01,908-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6620d3b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:10:01,911-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6620d3b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6620d3b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:10:01,914-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5d8256f3
>2018-05-30 02:10:01,914-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:10:02,014-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:10:02,014-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:10:02,014-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:10:02,014-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:10:02,014-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5d8256f3
>2018-05-30 02:10:02,014-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:10:02,027-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [3d2eb2b7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:10:02,028-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3d2eb2b7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61dcd41c
>2018-05-30 02:10:02,031-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3d2eb2b7] FINISH, SetVdsStatusVDSCommand, log id: 61dcd41c
>2018-05-30 02:10:02,036-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d2eb2b7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3d2eb2b7, Job ID: e903a2d3-d4f1-4434-b53b-cdcdfa78e9ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:10:02,041-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d2eb2b7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 384cba1d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:10:02,043-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d2eb2b7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 384cba1d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:10:02,053-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [3ebf9976] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:10:02,055-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [3ebf9976] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [4532c78d] Autorecovering 2 hosts
>2018-05-30 02:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [4532c78d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:15:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73920b12] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:15:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73920b12] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:15:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73920b12] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:15:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73920b12] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:15:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73920b12] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 175ac275
>2018-05-30 02:15:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73920b12] FINISH, SetVdsStatusVDSCommand, log id: 175ac275
>2018-05-30 02:15:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73920b12] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:15:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73920b12] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:15:00,082-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [73920b12] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:15:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [259255be] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:15:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [259255be] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:15:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [259255be] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:15:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [259255be] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:15:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [259255be] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 21d20439
>2018-05-30 02:15:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [259255be] FINISH, SetVdsStatusVDSCommand, log id: 21d20439
>2018-05-30 02:15:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [259255be] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3ae52e0d
>2018-05-30 02:15:00,103-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [259255be] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3ae52e0d
>2018-05-30 02:15:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [259255be] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:15:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [259255be] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:15:02,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2d74167] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: e2d5f99
>2018-05-30 02:15:02,516-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2d74167] FINISH, GetHardwareInfoVDSCommand, log id: e2d5f99
>2018-05-30 02:15:02,531-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6fcc50db] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:15:02,539-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [1b863745] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:15:02,544-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [8485ed1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:15:02,556-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [8485ed1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e559967
>2018-05-30 02:15:02,732-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [723d919f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1c42f9a8
>2018-05-30 02:15:02,734-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [723d919f] FINISH, GetHardwareInfoVDSCommand, log id: 1c42f9a8
>2018-05-30 02:15:02,746-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [c2799bb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:15:02,750-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [4b9e6904] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:15:02,756-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [9dc0709] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:15:02,761-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [9dc0709] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6711af5b
>2018-05-30 02:15:02,934-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8485ed1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:15:02,934-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [8485ed1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e559967
>2018-05-30 02:15:02,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8485ed1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:15:02,939-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [8485ed1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:15:02,943-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8485ed1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:15:02,944-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [8485ed1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:15:02,948-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8485ed1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:15:02,949-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [8485ed1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:15:02,951-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [8485ed1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:15:02,954-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8485ed1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 8485ed1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:15:02,961-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 41458143
>2018-05-30 02:15:02,961-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:15:03,059-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:15:03,059-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:15:03,059-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:15:03,059-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:15:03,059-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 41458143
>2018-05-30 02:15:03,060-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:15:03,073-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [6b3d0cf1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:15:03,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6b3d0cf1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25a9e913
>2018-05-30 02:15:03,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6b3d0cf1] FINISH, SetVdsStatusVDSCommand, log id: 25a9e913
>2018-05-30 02:15:03,108-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6b3d0cf1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6b3d0cf1, Job ID: 45ea22f6-caf8-41e7-8804-eaa857159ec8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:15:03,113-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6b3d0cf1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1b863745, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:15:03,115-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6b3d0cf1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1b863745, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:15:03,127-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [215a5091] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:15:03,128-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [215a5091] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:15:05,076-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9dc0709] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:15:05,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [9dc0709] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6711af5b
>2018-05-30 02:15:05,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9dc0709] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:15:05,081-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [9dc0709] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:15:05,085-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9dc0709] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:15:05,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [9dc0709] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:15:05,090-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9dc0709] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:15:05,091-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [9dc0709] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:15:05,094-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [9dc0709] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:15:05,097-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9dc0709] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 9dc0709, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:15:05,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 659fecd8
>2018-05-30 02:15:05,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:15:05,196-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:15:05,196-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:15:05,196-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:15:05,196-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:15:05,196-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 659fecd8
>2018-05-30 02:15:05,196-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:15:05,210-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [1bd91343] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:15:05,211-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1bd91343] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b34edd8
>2018-05-30 02:15:05,214-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1bd91343] FINISH, SetVdsStatusVDSCommand, log id: 4b34edd8
>2018-05-30 02:15:05,220-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1bd91343] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1bd91343, Job ID: c0a61f8e-1299-4bfe-89fe-1bf72020df3d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:15:05,225-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1bd91343] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4b9e6904, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:15:05,227-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1bd91343] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4b9e6904, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:15:05,238-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [1d9978ae] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:15:05,239-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [1d9978ae] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [369d31b8] Autorecovering 2 hosts
>2018-05-30 02:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [369d31b8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:20:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73147984] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:20:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73147984] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:20:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73147984] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:20:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73147984] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:20:00,047-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73147984] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 638c3af0
>2018-05-30 02:20:00,051-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73147984] FINISH, SetVdsStatusVDSCommand, log id: 638c3af0
>2018-05-30 02:20:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73147984] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:20:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73147984] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:20:00,054-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [73147984] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:20:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4961693d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:20:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4961693d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:20:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4961693d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:20:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4961693d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:20:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4961693d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 79223b6f
>2018-05-30 02:20:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4961693d] FINISH, SetVdsStatusVDSCommand, log id: 79223b6f
>2018-05-30 02:20:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [4961693d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 39016ee7
>2018-05-30 02:20:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [4961693d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 39016ee7
>2018-05-30 02:20:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4961693d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:20:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4961693d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:20:00,711-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4532c69a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1df41eed
>2018-05-30 02:20:00,713-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4532c69a] FINISH, GetHardwareInfoVDSCommand, log id: 1df41eed
>2018-05-30 02:20:00,725-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [7afa845a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:20:00,734-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [35fcecb7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:20:00,739-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [63505b9b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:20:00,799-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [63505b9b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 13b2bc52
>2018-05-30 02:20:01,185-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [63505b9b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:20:01,185-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [63505b9b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 13b2bc52
>2018-05-30 02:20:01,190-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [63505b9b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:20:01,190-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [63505b9b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:20:01,194-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [63505b9b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:20:01,195-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [63505b9b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:20:01,199-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [63505b9b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:20:01,200-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [63505b9b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:20:01,202-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [63505b9b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:20:01,205-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [63505b9b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 63505b9b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:20:01,211-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1e212bb9
>2018-05-30 02:20:01,211-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:20:01,309-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:20:01,309-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:20:01,309-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:20:01,309-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:20:01,309-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1e212bb9
>2018-05-30 02:20:01,309-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:20:01,322-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [36bc8106] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:20:01,324-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [36bc8106] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 20523b72
>2018-05-30 02:20:01,326-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [36bc8106] FINISH, SetVdsStatusVDSCommand, log id: 20523b72
>2018-05-30 02:20:01,334-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36bc8106] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 36bc8106, Job ID: 6e1d8497-ce43-426f-acbf-42f4e296de15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:20:01,338-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36bc8106] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 35fcecb7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:20:01,340-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36bc8106] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 35fcecb7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:20:01,352-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [78240ef] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:20:01,353-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [78240ef] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:20:02,902-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2b28dbfe] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 46b63e00
>2018-05-30 02:20:02,904-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2b28dbfe] FINISH, GetHardwareInfoVDSCommand, log id: 46b63e00
>2018-05-30 02:20:02,918-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [46626a58] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:20:02,922-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [4de074b2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:20:02,928-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [11ecd4c6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:20:02,935-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [11ecd4c6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 648fae0e
>2018-05-30 02:20:03,304-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [11ecd4c6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:20:03,304-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [11ecd4c6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 648fae0e
>2018-05-30 02:20:03,308-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [11ecd4c6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:20:03,309-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [11ecd4c6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:20:03,313-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [11ecd4c6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:20:03,314-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [11ecd4c6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:20:03,318-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [11ecd4c6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:20:03,318-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [11ecd4c6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:20:03,321-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [11ecd4c6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:20:03,324-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [11ecd4c6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 11ecd4c6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:20:03,327-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2d2a016d
>2018-05-30 02:20:03,327-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:20:03,425-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:20:03,425-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:20:03,425-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:20:03,425-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:20:03,425-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2d2a016d
>2018-05-30 02:20:03,425-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:20:03,440-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [1151005b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:20:03,441-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1151005b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 137524ae
>2018-05-30 02:20:03,443-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1151005b] FINISH, SetVdsStatusVDSCommand, log id: 137524ae
>2018-05-30 02:20:03,477-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1151005b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1151005b, Job ID: 78395a27-8825-430e-90da-263fe8165db8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:20:03,482-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1151005b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4de074b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:20:03,485-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1151005b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4de074b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:20:03,496-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [7c83c5ab] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:20:03,497-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [7c83c5ab] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7bd8dada] Autorecovering 2 hosts
>2018-05-30 02:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7bd8dada] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:25:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5f78e91e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5f78e91e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5f78e91e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5f78e91e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:25:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5f78e91e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76743721
>2018-05-30 02:25:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5f78e91e] FINISH, SetVdsStatusVDSCommand, log id: 76743721
>2018-05-30 02:25:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5f78e91e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:25:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5f78e91e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:25:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5f78e91e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:25:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7403dddf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7403dddf] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7403dddf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7403dddf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:25:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7403dddf] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25d48867
>2018-05-30 02:25:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7403dddf] FINISH, SetVdsStatusVDSCommand, log id: 25d48867
>2018-05-30 02:25:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [7403dddf] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 52ec677c
>2018-05-30 02:25:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [7403dddf] FINISH, SetHaMaintenanceModeVDSCommand, log id: 52ec677c
>2018-05-30 02:25:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7403dddf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:25:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7403dddf] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:25:01,134-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [362fbf91] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 63ad69b1
>2018-05-30 02:25:01,135-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [362fbf91] FINISH, GetHardwareInfoVDSCommand, log id: 63ad69b1
>2018-05-30 02:25:01,149-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [73ebd26a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:25:01,158-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [646bb7f3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:25:01,163-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [7e32c74f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:25:01,174-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [7e32c74f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22e27c95
>2018-05-30 02:25:01,548-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7e32c74f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:25:01,548-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [7e32c74f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 22e27c95
>2018-05-30 02:25:01,552-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7e32c74f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:25:01,553-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [7e32c74f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:25:01,556-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7e32c74f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:25:01,557-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [7e32c74f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:25:01,561-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7e32c74f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:25:01,562-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [7e32c74f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:25:01,564-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [7e32c74f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:25:01,567-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7e32c74f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7e32c74f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:25:01,573-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 43c1df7d
>2018-05-30 02:25:01,573-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:25:01,673-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:25:01,673-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:25:01,673-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:25:01,673-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:25:01,673-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 43c1df7d
>2018-05-30 02:25:01,673-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:25:01,686-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [54bcacff] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:25:01,687-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [54bcacff] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 35df8c69
>2018-05-30 02:25:01,689-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [54bcacff] FINISH, SetVdsStatusVDSCommand, log id: 35df8c69
>2018-05-30 02:25:01,722-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54bcacff] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 54bcacff, Job ID: a476d01c-3af2-4d80-908a-3506eecfe317, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:25:01,727-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54bcacff] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 646bb7f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:25:01,729-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54bcacff] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 646bb7f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:25:01,741-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [1513378a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:25:01,742-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1513378a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:25:01,965-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [12ebb2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6ed6a4a4
>2018-05-30 02:25:01,967-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [12ebb2] FINISH, GetHardwareInfoVDSCommand, log id: 6ed6a4a4
>2018-05-30 02:25:01,979-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [327b77bd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:25:01,983-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [d3317a2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:25:01,988-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [74ff964f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:25:01,995-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [74ff964f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 100439c5
>2018-05-30 02:25:02,373-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [74ff964f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:25:02,374-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [74ff964f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 100439c5
>2018-05-30 02:25:02,377-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [74ff964f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:25:02,378-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [74ff964f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:25:02,382-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [74ff964f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:25:02,383-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [74ff964f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:25:02,386-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [74ff964f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:25:02,387-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [74ff964f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:25:02,390-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [74ff964f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:25:02,393-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [74ff964f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 74ff964f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:25:02,396-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6a95d705
>2018-05-30 02:25:02,396-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:25:02,490-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:25:02,490-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:25:02,490-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:25:02,490-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:25:02,490-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6a95d705
>2018-05-30 02:25:02,490-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:25:02,502-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2fa787e3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:25:02,503-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2fa787e3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 315462f1
>2018-05-30 02:25:02,506-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2fa787e3] FINISH, SetVdsStatusVDSCommand, log id: 315462f1
>2018-05-30 02:25:02,511-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2fa787e3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2fa787e3, Job ID: 4c4b1dcd-d3b4-4430-8df8-e0de67ab1e4f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:25:02,516-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2fa787e3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: d3317a2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:25:02,518-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2fa787e3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: d3317a2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:25:02,528-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [285a5b02] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:25:02,529-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [285a5b02] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:30:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4b0e1fc8] Autorecovering 2 hosts
>2018-05-30 02:30:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4b0e1fc8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:30:00,011-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2e161e18] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:30:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2e161e18] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:30:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2e161e18] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:30:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2e161e18] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:30:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2e161e18] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3effd5c3
>2018-05-30 02:30:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2e161e18] FINISH, SetVdsStatusVDSCommand, log id: 3effd5c3
>2018-05-30 02:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2e161e18] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:30:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2e161e18] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:30:00,073-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2e161e18] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:30:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [555fae6b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:30:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [555fae6b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:30:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [555fae6b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:30:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [555fae6b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:30:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [555fae6b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12bc20ba
>2018-05-30 02:30:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [555fae6b] FINISH, SetVdsStatusVDSCommand, log id: 12bc20ba
>2018-05-30 02:30:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [555fae6b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 79e4e426
>2018-05-30 02:30:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [555fae6b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 79e4e426
>2018-05-30 02:30:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [555fae6b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:30:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [555fae6b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:30:03,159-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [56d875ad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: eb6147f
>2018-05-30 02:30:03,161-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [56d875ad] FINISH, GetHardwareInfoVDSCommand, log id: eb6147f
>2018-05-30 02:30:03,175-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [323dadeb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:30:03,184-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [79529eb0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:30:03,189-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [6294a7e7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:30:03,200-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [6294a7e7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3443979e
>2018-05-30 02:30:03,582-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6294a7e7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:30:03,582-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [6294a7e7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3443979e
>2018-05-30 02:30:03,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6294a7e7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:30:03,587-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [6294a7e7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:30:03,591-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6294a7e7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:30:03,591-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [6294a7e7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:30:03,595-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6294a7e7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:30:03,596-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [6294a7e7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:30:03,599-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [6294a7e7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:30:03,601-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6294a7e7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6294a7e7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:30:03,608-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 244bdd88
>2018-05-30 02:30:03,608-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:30:03,707-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:30:03,707-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:30:03,707-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:30:03,707-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:30:03,707-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 244bdd88
>2018-05-30 02:30:03,707-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:30:03,721-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [79bd7edb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:30:03,722-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [79bd7edb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52518b04
>2018-05-30 02:30:03,724-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [79bd7edb] FINISH, SetVdsStatusVDSCommand, log id: 52518b04
>2018-05-30 02:30:03,755-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [79bd7edb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 79bd7edb, Job ID: 30491a8b-4be3-4649-a593-3b1537ceabd1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:30:03,760-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [79bd7edb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 79529eb0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:30:03,762-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [79bd7edb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 79529eb0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:30:03,773-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [66cf2d52] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:30:03,774-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [66cf2d52] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:30:03,938-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6a5b5f73] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5872be2e
>2018-05-30 02:30:03,940-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6a5b5f73] FINISH, GetHardwareInfoVDSCommand, log id: 5872be2e
>2018-05-30 02:30:03,954-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [7fed691a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:30:03,958-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [10feb5ab] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:30:03,964-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [21a526e4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:30:03,971-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [21a526e4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 59ac93f0
>2018-05-30 02:30:04,344-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [21a526e4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:30:04,344-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [21a526e4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 59ac93f0
>2018-05-30 02:30:04,348-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [21a526e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:30:04,349-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [21a526e4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:30:04,352-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [21a526e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:30:04,353-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [21a526e4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:30:04,357-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [21a526e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:30:04,358-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [21a526e4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:30:04,361-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [21a526e4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:30:04,364-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [21a526e4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 21a526e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:30:04,388-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 607f211a
>2018-05-30 02:30:04,388-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:30:04,487-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:30:04,487-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:30:04,488-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:30:04,488-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:30:04,488-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 607f211a
>2018-05-30 02:30:04,488-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:30:04,502-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [3c319c50] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:30:04,503-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3c319c50] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2044d4f2
>2018-05-30 02:30:04,506-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3c319c50] FINISH, SetVdsStatusVDSCommand, log id: 2044d4f2
>2018-05-30 02:30:04,512-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3c319c50] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c319c50, Job ID: b4545efa-cca4-4b79-ab2d-51e0fe6374a6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:30:04,517-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3c319c50] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 10feb5ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:30:04,519-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3c319c50] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 10feb5ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:30:04,530-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [55fd9eaf] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:30:04,531-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [55fd9eaf] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:35:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6d29498e] Autorecovering 2 hosts
>2018-05-30 02:35:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6d29498e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:35:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f1b4558] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:35:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f1b4558] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:35:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f1b4558] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:35:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f1b4558] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:35:00,043-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6f1b4558] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c5be0e
>2018-05-30 02:35:00,045-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6f1b4558] FINISH, SetVdsStatusVDSCommand, log id: 7c5be0e
>2018-05-30 02:35:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f1b4558] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:35:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f1b4558] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:35:00,048-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6f1b4558] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:35:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [42c88103] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:35:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [42c88103] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:35:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [42c88103] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:35:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [42c88103] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:35:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [42c88103] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2997975e
>2018-05-30 02:35:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [42c88103] FINISH, SetVdsStatusVDSCommand, log id: 2997975e
>2018-05-30 02:35:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [42c88103] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 65d7533f
>2018-05-30 02:35:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [42c88103] FINISH, SetHaMaintenanceModeVDSCommand, log id: 65d7533f
>2018-05-30 02:35:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [42c88103] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [42c88103] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:35:01,429-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [512a0803] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4215d810
>2018-05-30 02:35:01,431-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [512a0803] FINISH, GetHardwareInfoVDSCommand, log id: 4215d810
>2018-05-30 02:35:01,444-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [47eed417] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:35:01,448-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [997b800] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:35:01,453-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [682c3c01] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:35:01,465-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [682c3c01] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7a01a68e
>2018-05-30 02:35:01,839-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [682c3c01] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:35:01,839-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [682c3c01] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7a01a68e
>2018-05-30 02:35:01,843-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [682c3c01] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:35:01,844-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [682c3c01] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:35:01,848-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [682c3c01] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:35:01,848-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [682c3c01] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:35:01,852-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [682c3c01] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:35:01,853-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [682c3c01] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:35:01,855-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [682c3c01] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:35:01,858-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [682c3c01] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 682c3c01, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:35:01,865-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5955ff18
>2018-05-30 02:35:01,865-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:35:01,964-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:35:01,964-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:35:01,964-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:35:01,964-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:35:01,964-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5955ff18
>2018-05-30 02:35:01,964-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:35:01,978-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [3f7d1313] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:35:01,979-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3f7d1313] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b5bca2e
>2018-05-30 02:35:01,982-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3f7d1313] FINISH, SetVdsStatusVDSCommand, log id: 4b5bca2e
>2018-05-30 02:35:01,990-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f7d1313] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3f7d1313, Job ID: 75c8d9b1-9f6e-48b9-b599-807ed0f85ed4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:35:01,994-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f7d1313] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 997b800, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:35:01,996-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f7d1313] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 997b800, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:35:02,007-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [1b437dd] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:35:02,008-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1b437dd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:35:02,126-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [22ac59da] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 18610499
>2018-05-30 02:35:02,128-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [22ac59da] FINISH, GetHardwareInfoVDSCommand, log id: 18610499
>2018-05-30 02:35:02,140-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [7ceeda5f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:35:02,145-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [44f8acca] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:35:02,150-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [4d113ac2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:35:02,205-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [4d113ac2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 18f217a
>2018-05-30 02:35:02,590-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4d113ac2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:35:02,590-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [4d113ac2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 18f217a
>2018-05-30 02:35:02,594-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4d113ac2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:35:02,595-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [4d113ac2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:35:02,599-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4d113ac2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:35:02,599-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [4d113ac2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:35:02,603-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4d113ac2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:35:02,604-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [4d113ac2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:35:02,607-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [4d113ac2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:35:02,609-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4d113ac2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d113ac2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:35:02,616-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 22a64c2f
>2018-05-30 02:35:02,616-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:35:02,715-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:35:02,715-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:35:02,715-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:35:02,715-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:35:02,715-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 22a64c2f
>2018-05-30 02:35:02,715-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:35:02,729-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [4e0e7201] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:35:02,730-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4e0e7201] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4926bffb
>2018-05-30 02:35:02,733-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4e0e7201] FINISH, SetVdsStatusVDSCommand, log id: 4926bffb
>2018-05-30 02:35:02,741-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4e0e7201] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4e0e7201, Job ID: d672bb8a-8b6c-4ac9-83a9-362dcad63cc2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:35:02,745-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4e0e7201] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 44f8acca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:35:02,748-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4e0e7201] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 44f8acca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:35:02,758-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [a6f9015] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:35:02,759-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [a6f9015] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7dfb874c] Autorecovering 2 hosts
>2018-05-30 02:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7dfb874c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:40:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [36eac085] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [36eac085] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [36eac085] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:40:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [36eac085] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:40:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [36eac085] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39eaf271
>2018-05-30 02:40:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [36eac085] FINISH, SetVdsStatusVDSCommand, log id: 39eaf271
>2018-05-30 02:40:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [36eac085] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:40:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [36eac085] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:40:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [36eac085] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:40:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4052a812] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4052a812] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4052a812] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4052a812] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:40:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4052a812] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45a0041
>2018-05-30 02:40:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4052a812] FINISH, SetVdsStatusVDSCommand, log id: 45a0041
>2018-05-30 02:40:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [4052a812] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 331167df
>2018-05-30 02:40:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [4052a812] FINISH, SetHaMaintenanceModeVDSCommand, log id: 331167df
>2018-05-30 02:40:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4052a812] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:40:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4052a812] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:40:00,410-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [6f16b5f9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6c4844b
>2018-05-30 02:40:00,411-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [6f16b5f9] FINISH, GetHardwareInfoVDSCommand, log id: 6c4844b
>2018-05-30 02:40:00,424-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [3704ab0d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:40:00,433-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [600b1ce7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:40:00,438-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [422fdd6c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:40:00,450-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [422fdd6c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7e5f38b6
>2018-05-30 02:40:00,826-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [422fdd6c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:40:00,826-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [422fdd6c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7e5f38b6
>2018-05-30 02:40:00,830-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [422fdd6c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:40:00,830-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [422fdd6c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:40:00,834-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [422fdd6c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:40:00,835-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [422fdd6c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:40:00,839-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [422fdd6c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:40:00,840-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [422fdd6c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:40:00,843-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [422fdd6c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:40:00,846-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [422fdd6c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 422fdd6c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:40:00,853-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: bf5addf
>2018-05-30 02:40:00,853-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:40:00,947-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:40:00,947-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:40:00,947-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:40:00,947-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:40:00,947-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: bf5addf
>2018-05-30 02:40:00,947-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:40:00,962-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [15986d2f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:40:00,963-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [15986d2f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2b105066
>2018-05-30 02:40:00,965-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [15986d2f] FINISH, SetVdsStatusVDSCommand, log id: 2b105066
>2018-05-30 02:40:00,998-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [15986d2f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 15986d2f, Job ID: 55546093-d4a6-4522-803a-507f3f0bff00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:40:01,003-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [15986d2f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 600b1ce7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:40:01,006-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [15986d2f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 600b1ce7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:40:01,016-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [382e8f14] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:40:01,017-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [382e8f14] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:40:03,007-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [688d28cf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 71d74f12
>2018-05-30 02:40:03,009-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [688d28cf] FINISH, GetHardwareInfoVDSCommand, log id: 71d74f12
>2018-05-30 02:40:03,023-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [63108e39] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:40:03,027-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [4be007f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:40:03,032-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5f317b7d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:40:03,039-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5f317b7d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4f92d7f9
>2018-05-30 02:40:03,409-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5f317b7d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:40:03,409-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5f317b7d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4f92d7f9
>2018-05-30 02:40:03,413-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5f317b7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:40:03,414-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5f317b7d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:40:03,417-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5f317b7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:40:03,418-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5f317b7d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:40:03,422-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5f317b7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:40:03,423-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5f317b7d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:40:03,426-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5f317b7d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:40:03,428-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5f317b7d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5f317b7d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:40:03,431-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6a86b68
>2018-05-30 02:40:03,431-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:40:03,529-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:40:03,529-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:40:03,529-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:40:03,529-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:40:03,529-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6a86b68
>2018-05-30 02:40:03,529-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:40:03,542-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [524f0417] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:40:03,544-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [524f0417] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f512a13
>2018-05-30 02:40:03,546-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [524f0417] FINISH, SetVdsStatusVDSCommand, log id: 3f512a13
>2018-05-30 02:40:03,551-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [524f0417] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 524f0417, Job ID: feb96d6a-23c1-46a8-bfa5-cbbeeb34ad1e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:40:03,556-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [524f0417] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4be007f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:40:03,558-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [524f0417] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4be007f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:40:03,569-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [36405e00] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:40:03,570-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [36405e00] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [761b134d] Autorecovering 2 hosts
>2018-05-30 02:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [761b134d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:45:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [21b5232e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:45:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [21b5232e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:45:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [21b5232e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:45:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [21b5232e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:45:00,052-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [21b5232e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43696b08
>2018-05-30 02:45:00,055-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [21b5232e] FINISH, SetVdsStatusVDSCommand, log id: 43696b08
>2018-05-30 02:45:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [21b5232e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:45:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [21b5232e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:45:00,058-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [21b5232e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:45:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [fe01a8e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:45:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [fe01a8e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:45:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [fe01a8e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:45:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [fe01a8e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:45:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [fe01a8e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 37095e8
>2018-05-30 02:45:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [fe01a8e] FINISH, SetVdsStatusVDSCommand, log id: 37095e8
>2018-05-30 02:45:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [fe01a8e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6e18ff9b
>2018-05-30 02:45:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [fe01a8e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6e18ff9b
>2018-05-30 02:45:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [fe01a8e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:45:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [fe01a8e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:45:01,300-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [62221bff] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7d5f9b6f
>2018-05-30 02:45:01,302-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [62221bff] FINISH, GetHardwareInfoVDSCommand, log id: 7d5f9b6f
>2018-05-30 02:45:01,315-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [5e179800] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:45:01,323-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [1c2e26df] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:45:01,329-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [63c7df41] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:45:01,335-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [63c7df41] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 562e6e07
>2018-05-30 02:45:01,608-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4253d159] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 68dd1db4
>2018-05-30 02:45:01,610-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4253d159] FINISH, GetHardwareInfoVDSCommand, log id: 68dd1db4
>2018-05-30 02:45:01,623-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [59b86d2d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:45:01,627-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [1637deeb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:45:01,632-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [592e8109] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:45:01,638-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [592e8109] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7635ea62
>2018-05-30 02:45:01,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63c7df41] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:45:01,704-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [63c7df41] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 562e6e07
>2018-05-30 02:45:01,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63c7df41] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:45:01,709-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [63c7df41] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:45:01,713-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63c7df41] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:45:01,714-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [63c7df41] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:45:01,718-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63c7df41] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:45:01,719-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [63c7df41] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:45:01,721-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [63c7df41] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:45:01,724-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63c7df41] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 63c7df41, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:45:01,727-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2ef3c8fb
>2018-05-30 02:45:01,727-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:45:01,859-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:45:01,859-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:45:01,859-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:45:01,859-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:45:01,859-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2ef3c8fb
>2018-05-30 02:45:01,859-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:45:01,872-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [fdf8128] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:45:01,873-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [fdf8128] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 564394bc
>2018-05-30 02:45:01,875-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [fdf8128] FINISH, SetVdsStatusVDSCommand, log id: 564394bc
>2018-05-30 02:45:01,905-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [fdf8128] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: fdf8128, Job ID: 4cc9f5e4-049c-4bec-a516-fd7573902482, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:45:01,910-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [fdf8128] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1c2e26df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:45:01,912-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [fdf8128] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1c2e26df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:45:01,923-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [550b1a0e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:45:01,924-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [550b1a0e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:45:02,013-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [592e8109] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:45:02,013-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [592e8109] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7635ea62
>2018-05-30 02:45:02,017-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [592e8109] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:45:02,018-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [592e8109] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:45:02,022-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [592e8109] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:45:02,023-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [592e8109] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:45:02,026-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [592e8109] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:45:02,027-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [592e8109] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:45:02,030-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [592e8109] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:45:02,033-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [592e8109] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 592e8109, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:45:02,036-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3abdb66
>2018-05-30 02:45:02,036-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:45:02,134-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:45:02,134-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:45:02,134-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:45:02,134-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:45:02,134-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3abdb66
>2018-05-30 02:45:02,134-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:45:02,147-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [40a7abd2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:45:02,148-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [40a7abd2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52814f78
>2018-05-30 02:45:02,150-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [40a7abd2] FINISH, SetVdsStatusVDSCommand, log id: 52814f78
>2018-05-30 02:45:02,156-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [40a7abd2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 40a7abd2, Job ID: b7f8cb60-25a4-416a-86ab-417e81fb414f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:45:02,161-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [40a7abd2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1637deeb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:45:02,163-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [40a7abd2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1637deeb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:45:02,173-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [7dcc90b8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:45:02,174-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7dcc90b8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7df155d0] Autorecovering 2 hosts
>2018-05-30 02:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7df155d0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:50:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c64aaf7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:50:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c64aaf7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:50:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c64aaf7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:50:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c64aaf7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:50:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [c64aaf7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: e7f56b7
>2018-05-30 02:50:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [c64aaf7] FINISH, SetVdsStatusVDSCommand, log id: e7f56b7
>2018-05-30 02:50:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c64aaf7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:50:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c64aaf7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:50:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [c64aaf7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:50:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30267ffb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:50:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30267ffb] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:50:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30267ffb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:50:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30267ffb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:50:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [30267ffb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3408ea38
>2018-05-30 02:50:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [30267ffb] FINISH, SetVdsStatusVDSCommand, log id: 3408ea38
>2018-05-30 02:50:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [30267ffb] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6894063b
>2018-05-30 02:50:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [30267ffb] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6894063b
>2018-05-30 02:50:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30267ffb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:50:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30267ffb] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:50:02,740-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [54162563] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 52cb89f6
>2018-05-30 02:50:02,742-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [54162563] FINISH, GetHardwareInfoVDSCommand, log id: 52cb89f6
>2018-05-30 02:50:02,748-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5e48d348] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4fd8d6d
>2018-05-30 02:50:02,750-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5e48d348] FINISH, GetHardwareInfoVDSCommand, log id: 4fd8d6d
>2018-05-30 02:50:02,758-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [52321471] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:50:02,767-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4e49e17a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:50:02,772-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [43de80ca] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:50:02,780-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [65df50ea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:50:02,788-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [f1fdf12] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:50:02,794-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7f6bcefd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:50:02,838-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [43de80ca] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4e90a999
>2018-05-30 02:50:02,852-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7f6bcefd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 79dbe611
>2018-05-30 02:50:03,229-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f6bcefd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:50:03,229-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7f6bcefd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 79dbe611
>2018-05-30 02:50:03,232-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [43de80ca] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:50:03,232-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [43de80ca] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4e90a999
>2018-05-30 02:50:03,236-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f6bcefd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:50:03,236-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7f6bcefd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:50:03,239-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [43de80ca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:50:03,240-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [43de80ca] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:50:03,243-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f6bcefd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:50:03,244-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7f6bcefd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:50:03,244-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [43de80ca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:50:03,245-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [43de80ca] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:50:03,248-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f6bcefd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:50:03,249-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7f6bcefd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:50:03,250-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [43de80ca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:50:03,250-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [43de80ca] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:50:03,253-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7f6bcefd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:50:03,254-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [43de80ca] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:50:03,256-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f6bcefd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7f6bcefd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:50:03,257-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [43de80ca] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 43de80ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:50:03,262-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 41f60b7a
>2018-05-30 02:50:03,262-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:50:03,360-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:50:03,360-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:50:03,360-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:50:03,360-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:50:03,360-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 41f60b7a
>2018-05-30 02:50:03,360-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:50:03,363-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5d81519c
>2018-05-30 02:50:03,363-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:50:03,375-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [2b2755be] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:50:03,376-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2b2755be] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 402c250a
>2018-05-30 02:50:03,378-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2b2755be] FINISH, SetVdsStatusVDSCommand, log id: 402c250a
>2018-05-30 02:50:03,410-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2b2755be] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2b2755be, Job ID: 9c4693ce-4366-4aa8-906b-d3af1cf170fa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:50:03,415-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2b2755be] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: f1fdf12, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:50:03,418-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2b2755be] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: f1fdf12, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:50:03,430-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [5adfeee3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:50:03,431-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [5adfeee3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:50:03,461-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:50:03,461-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:50:03,462-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:50:03,462-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:50:03,462-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5d81519c
>2018-05-30 02:50:03,462-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:50:03,476-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [3497e10d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:50:03,477-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3497e10d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6015d021
>2018-05-30 02:50:03,479-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3497e10d] FINISH, SetVdsStatusVDSCommand, log id: 6015d021
>2018-05-30 02:50:03,512-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3497e10d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3497e10d, Job ID: 8abb2902-fad5-452b-833d-72a9c45625e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:50:03,517-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3497e10d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4e49e17a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:50:03,519-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3497e10d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4e49e17a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:50:03,531-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [5471bbe] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:50:03,532-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [5471bbe] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:54:43,472-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler5) [2e9b92f3] Attempting to update VMs/Templates Ovf.
>2018-05-30 02:54:43,475-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 02:54:43,475-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 02:54:43,476-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:54:43,480-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-30 02:54:43,483-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-30 02:54:43,483-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Attempting to update template OVFs in Data Center 'Default'
>2018-05-30 02:54:43,483-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-30 02:54:43,483-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 02:54:43,484-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 02:54:43,484-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 02:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [2e344ff3] Autorecovering 2 hosts
>2018-05-30 02:55:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [2e344ff3] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:55:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [e1f6340] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [e1f6340] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [e1f6340] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:55:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [e1f6340] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:55:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [e1f6340] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33264a07
>2018-05-30 02:55:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [e1f6340] FINISH, SetVdsStatusVDSCommand, log id: 33264a07
>2018-05-30 02:55:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [e1f6340] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:55:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [e1f6340] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 02:55:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [e1f6340] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:55:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5409ed5b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:55:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5409ed5b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 02:55:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5409ed5b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:55:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5409ed5b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:55:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5409ed5b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4723f6d9
>2018-05-30 02:55:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5409ed5b] FINISH, SetVdsStatusVDSCommand, log id: 4723f6d9
>2018-05-30 02:55:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5409ed5b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 22d033bd
>2018-05-30 02:55:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5409ed5b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 22d033bd
>2018-05-30 02:55:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5409ed5b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 02:55:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5409ed5b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 02:55:01,069-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [ca53216] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 75b96997
>2018-05-30 02:55:01,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [ca53216] FINISH, GetHardwareInfoVDSCommand, log id: 75b96997
>2018-05-30 02:55:01,083-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [131cdeed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:55:01,083-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4060913b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: d047f2
>2018-05-30 02:55:01,085-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4060913b] FINISH, GetHardwareInfoVDSCommand, log id: d047f2
>2018-05-30 02:55:01,089-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [7108be4f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:55:01,095-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5668ac0f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:55:01,101-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7a68c411] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:55:01,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5668ac0f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4e9f095c
>2018-05-30 02:55:01,106-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [4826057b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:55:01,111-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5d2516ee] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 02:55:01,117-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5d2516ee] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f3a9398
>2018-05-30 02:55:01,482-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5668ac0f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:55:01,482-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5668ac0f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4e9f095c
>2018-05-30 02:55:01,486-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5668ac0f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:55:01,487-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5668ac0f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:55:01,491-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5668ac0f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:55:01,491-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5668ac0f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:55:01,492-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5d2516ee] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 02:55:01,492-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5d2516ee] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6f3a9398
>2018-05-30 02:55:01,495-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5668ac0f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:55:01,496-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5668ac0f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:55:01,496-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5d2516ee] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:55:01,497-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5d2516ee] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:55:01,499-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5668ac0f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:55:01,501-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5d2516ee] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:55:01,502-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5d2516ee] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:55:01,502-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5668ac0f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5668ac0f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:55:01,505-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 75f164a3
>2018-05-30 02:55:01,505-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:55:01,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5d2516ee] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 02:55:01,508-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5d2516ee] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 02:55:01,511-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5d2516ee] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 02:55:01,513-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5d2516ee] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5d2516ee, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 02:55:01,602-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:55:01,602-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:55:01,602-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:55:01,602-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:55:01,602-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 75f164a3
>2018-05-30 02:55:01,602-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:55:01,607-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 377ca585
>2018-05-30 02:55:01,607-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 02:55:01,615-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [228d5db6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:55:01,616-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [228d5db6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 62581e9e
>2018-05-30 02:55:01,619-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [228d5db6] FINISH, SetVdsStatusVDSCommand, log id: 62581e9e
>2018-05-30 02:55:01,624-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [228d5db6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 228d5db6, Job ID: 410db48d-f282-408f-89a5-8b524f9ae6ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:55:01,629-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [228d5db6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7108be4f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:55:01,631-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [228d5db6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7108be4f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:55:01,641-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [5a78f420] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 02:55:01,643-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [5a78f420] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 02:55:01,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:55:01,708-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 02:55:01,708-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 02:55:01,708-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 02:55:01,708-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 377ca585
>2018-05-30 02:55:01,708-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 02:55:01,721-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [73a34712] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:55:01,722-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [73a34712] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d518416
>2018-05-30 02:55:01,724-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [73a34712] FINISH, SetVdsStatusVDSCommand, log id: 3d518416
>2018-05-30 02:55:01,729-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [73a34712] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 73a34712, Job ID: 784fe7d7-04d2-4a78-9a7f-6eebb2c43b70, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 02:55:01,734-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [73a34712] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4826057b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 02:55:01,736-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [73a34712] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4826057b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 02:55:01,746-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [41b332f8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 02:55:01,747-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [41b332f8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:00:00,006-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [1113e8ad] Autorecovering 2 hosts
>2018-05-30 03:00:00,006-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [1113e8ad] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:00:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [545f3d61] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:00:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [545f3d61] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:00:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [545f3d61] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:00:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [545f3d61] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:00:00,046-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [545f3d61] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 28a61d18
>2018-05-30 03:00:00,048-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [545f3d61] FINISH, SetVdsStatusVDSCommand, log id: 28a61d18
>2018-05-30 03:00:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [545f3d61] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:00:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [545f3d61] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:00:00,051-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [545f3d61] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:00:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [428a84ed] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:00:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [428a84ed] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:00:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [428a84ed] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:00:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [428a84ed] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:00:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [428a84ed] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7057c6cc
>2018-05-30 03:00:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [428a84ed] FINISH, SetVdsStatusVDSCommand, log id: 7057c6cc
>2018-05-30 03:00:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [428a84ed] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1b3520a3
>2018-05-30 03:00:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [428a84ed] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1b3520a3
>2018-05-30 03:00:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [428a84ed] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:00:00,076-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [428a84ed] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 428a84ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-30 03:00:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [428a84ed] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:00:02,247-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [65c348a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7c4cffc6
>2018-05-30 03:00:02,249-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [65c348a] FINISH, GetHardwareInfoVDSCommand, log id: 7c4cffc6
>2018-05-30 03:00:02,263-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [4ac64f9e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:00:02,268-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [75a7fab3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:00:02,273-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3e73de89] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:00:02,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3e73de89] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2cdb70f1
>2018-05-30 03:00:02,494-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [5cec2a64] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 12c8e62c
>2018-05-30 03:00:02,496-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [5cec2a64] FINISH, GetHardwareInfoVDSCommand, log id: 12c8e62c
>2018-05-30 03:00:02,508-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [7a2004f1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:00:02,513-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [251412e8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:00:02,518-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [214c2296] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:00:02,524-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [214c2296] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 712bb4f0
>2018-05-30 03:00:02,658-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e73de89] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:00:02,658-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3e73de89] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2cdb70f1
>2018-05-30 03:00:02,662-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e73de89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:00:02,663-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e73de89] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:00:02,667-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e73de89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:00:02,668-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e73de89] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:00:02,671-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e73de89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:00:02,672-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e73de89] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:00:02,675-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3e73de89] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:00:02,678-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e73de89] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3e73de89, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:00:02,684-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a1d1d8
>2018-05-30 03:00:02,684-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:00:02,779-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:00:02,779-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:00:02,779-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:00:02,779-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:00:02,779-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a1d1d8
>2018-05-30 03:00:02,779-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:00:02,793-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [f379df9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:00:02,794-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [f379df9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 796f9c62
>2018-05-30 03:00:02,797-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [f379df9] FINISH, SetVdsStatusVDSCommand, log id: 796f9c62
>2018-05-30 03:00:02,805-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [f379df9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: f379df9, Job ID: d84721d3-550b-42af-9321-45c4cee8bda6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:00:02,810-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [f379df9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 75a7fab3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:00:02,812-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [f379df9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 75a7fab3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:00:02,823-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [50c8db07] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:00:02,824-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [50c8db07] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:00:02,891-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [214c2296] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:00:02,891-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [214c2296] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 712bb4f0
>2018-05-30 03:00:02,895-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [214c2296] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:00:02,896-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [214c2296] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:00:02,900-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [214c2296] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:00:02,900-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [214c2296] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:00:02,904-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [214c2296] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:00:02,905-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [214c2296] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:00:02,908-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [214c2296] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:00:02,911-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [214c2296] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 214c2296, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:00:02,913-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 47e5d083
>2018-05-30 03:00:02,913-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:00:03,011-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:00:03,011-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:00:03,011-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:00:03,011-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:00:03,011-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 47e5d083
>2018-05-30 03:00:03,011-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:00:03,023-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [186eb74b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:00:03,025-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [186eb74b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17a57bcb
>2018-05-30 03:00:03,027-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [186eb74b] FINISH, SetVdsStatusVDSCommand, log id: 17a57bcb
>2018-05-30 03:00:03,032-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [186eb74b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 186eb74b, Job ID: 324765ea-b2e9-46e8-bca0-fb22b7859cb1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:00:03,037-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [186eb74b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 251412e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:00:03,039-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [186eb74b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 251412e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:00:03,049-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [9beb6dc] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:00:03,050-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [9beb6dc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [68ad7e16] Autorecovering 2 hosts
>2018-05-30 03:05:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [68ad7e16] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:05:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [34f1cf19] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:05:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [34f1cf19] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:05:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [34f1cf19] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:05:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [34f1cf19] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:05:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [34f1cf19] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2727ef33
>2018-05-30 03:05:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [34f1cf19] FINISH, SetVdsStatusVDSCommand, log id: 2727ef33
>2018-05-30 03:05:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [34f1cf19] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:05:00,076-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [34f1cf19] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 34f1cf19, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-30 03:05:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [34f1cf19] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:05:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [34f1cf19] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:05:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52cff895] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52cff895] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52cff895] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:05:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52cff895] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:05:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [52cff895] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 35af292c
>2018-05-30 03:05:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [52cff895] FINISH, SetVdsStatusVDSCommand, log id: 35af292c
>2018-05-30 03:05:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [52cff895] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 971a761
>2018-05-30 03:05:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [52cff895] FINISH, SetHaMaintenanceModeVDSCommand, log id: 971a761
>2018-05-30 03:05:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52cff895] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:05:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52cff895] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:05:00,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [5ca7dc28] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 742a82de
>2018-05-30 03:05:00,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [5ca7dc28] FINISH, GetHardwareInfoVDSCommand, log id: 742a82de
>2018-05-30 03:05:00,529-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [5d511f2c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:05:00,537-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5b25e2b2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:05:00,542-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [49881acf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:05:00,554-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [49881acf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6a1833c3
>2018-05-30 03:05:00,651-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3be9dacf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 17943e0b
>2018-05-30 03:05:00,652-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3be9dacf] FINISH, GetHardwareInfoVDSCommand, log id: 17943e0b
>2018-05-30 03:05:00,665-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6d1f6564] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:05:00,669-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [525f5094] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:05:00,675-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3bafa7ac] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:05:00,681-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3bafa7ac] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 52715ec7
>2018-05-30 03:05:00,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49881acf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:05:00,935-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [49881acf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6a1833c3
>2018-05-30 03:05:00,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49881acf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:05:00,940-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [49881acf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:05:00,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49881acf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:05:00,945-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [49881acf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:05:00,949-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49881acf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:05:00,950-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [49881acf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:05:00,953-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [49881acf] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:05:00,955-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49881acf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 49881acf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:05:00,962-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3ca70d6f
>2018-05-30 03:05:00,962-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:05:01,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3bafa7ac] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:05:01,052-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3bafa7ac] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 52715ec7
>2018-05-30 03:05:01,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3bafa7ac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:05:01,057-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3bafa7ac] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:05:01,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3bafa7ac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:05:01,062-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3bafa7ac] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:05:01,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:05:01,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:05:01,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:05:01,063-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:05:01,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3ca70d6f
>2018-05-30 03:05:01,063-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:05:01,066-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3bafa7ac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:05:01,067-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3bafa7ac] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:05:01,071-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3bafa7ac] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:05:01,073-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3bafa7ac] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3bafa7ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:05:01,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6b14d575
>2018-05-30 03:05:01,076-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:05:01,078-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [63fe8db1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:05:01,080-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [63fe8db1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31e1156b
>2018-05-30 03:05:01,082-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [63fe8db1] FINISH, SetVdsStatusVDSCommand, log id: 31e1156b
>2018-05-30 03:05:01,115-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63fe8db1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 63fe8db1, Job ID: 35ca4777-982a-416e-8250-b73482f796c6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:05:01,120-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63fe8db1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5b25e2b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:05:01,122-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63fe8db1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5b25e2b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:05:01,133-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [76501e05] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:05:01,134-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [76501e05] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:05:01,176-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:05:01,176-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:05:01,177-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:05:01,177-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:05:01,177-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6b14d575
>2018-05-30 03:05:01,177-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:05:01,190-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [29b5cf2b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:05:01,191-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [29b5cf2b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50f796b8
>2018-05-30 03:05:01,193-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [29b5cf2b] FINISH, SetVdsStatusVDSCommand, log id: 50f796b8
>2018-05-30 03:05:01,199-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [29b5cf2b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 29b5cf2b, Job ID: b333ab19-10e1-4cd0-a2c7-028a6ea6ad9e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:05:01,203-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [29b5cf2b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 525f5094, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:05:01,206-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [29b5cf2b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 525f5094, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:05:01,216-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [11f178b3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:05:01,217-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [11f178b3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:10:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [264bf43c] Autorecovering 2 hosts
>2018-05-30 03:10:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [264bf43c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:10:00,013-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1b972f6b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:10:00,019-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1b972f6b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:10:00,019-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1b972f6b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:10:00,019-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1b972f6b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:10:00,021-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1b972f6b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3033bfa0
>2018-05-30 03:10:00,023-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1b972f6b] FINISH, SetVdsStatusVDSCommand, log id: 3033bfa0
>2018-05-30 03:10:00,025-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1b972f6b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:10:00,026-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1b972f6b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:10:00,026-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1b972f6b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:10:00,032-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d941db5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:10:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d941db5] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:10:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d941db5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:10:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d941db5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:10:00,039-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1d941db5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13fc2b0b
>2018-05-30 03:10:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1d941db5] FINISH, SetVdsStatusVDSCommand, log id: 13fc2b0b
>2018-05-30 03:10:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [1d941db5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 61289c8f
>2018-05-30 03:10:00,046-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [1d941db5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 61289c8f
>2018-05-30 03:10:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d941db5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:10:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d941db5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:10:01,757-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5f8c67a3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 45ec3c27
>2018-05-30 03:10:01,759-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5f8c67a3] FINISH, GetHardwareInfoVDSCommand, log id: 45ec3c27
>2018-05-30 03:10:01,772-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [2a3c314e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:10:01,777-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [7eca0e17] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:10:01,782-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3b2ea602] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:10:01,790-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3b2ea602] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6daa8e5d
>2018-05-30 03:10:01,867-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [46140736] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1806e091
>2018-05-30 03:10:01,869-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [46140736] FINISH, GetHardwareInfoVDSCommand, log id: 1806e091
>2018-05-30 03:10:01,881-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [a2d3aa4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:10:01,886-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [2f84a16b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:10:01,891-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [42b511f9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:10:01,897-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [42b511f9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7a491e41
>2018-05-30 03:10:02,162-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b2ea602] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:10:02,162-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3b2ea602] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6daa8e5d
>2018-05-30 03:10:02,166-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b2ea602] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:10:02,167-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3b2ea602] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:10:02,170-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b2ea602] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:10:02,171-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3b2ea602] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:10:02,175-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b2ea602] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:10:02,176-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3b2ea602] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:10:02,179-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3b2ea602] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:10:02,182-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b2ea602] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3b2ea602, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:10:02,185-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 616345ac
>2018-05-30 03:10:02,185-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:10:02,267-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [42b511f9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:10:02,267-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [42b511f9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7a491e41
>2018-05-30 03:10:02,272-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [42b511f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:10:02,272-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [42b511f9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:10:02,276-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [42b511f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:10:02,277-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [42b511f9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:10:02,281-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:10:02,281-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:10:02,281-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:10:02,281-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:10:02,281-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 616345ac
>2018-05-30 03:10:02,281-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:10:02,282-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [42b511f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:10:02,283-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [42b511f9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:10:02,286-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [42b511f9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:10:02,289-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [42b511f9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 42b511f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:10:02,293-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [6e16e967] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:10:02,294-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6e16e967] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 252d4602
>2018-05-30 03:10:02,296-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3b7ebb04
>2018-05-30 03:10:02,296-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:10:02,297-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6e16e967] FINISH, SetVdsStatusVDSCommand, log id: 252d4602
>2018-05-30 03:10:02,329-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6e16e967] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6e16e967, Job ID: ace5c1bd-b936-45b3-815f-3b08192c4f56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:10:02,333-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6e16e967] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7eca0e17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:10:02,336-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6e16e967] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7eca0e17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:10:02,347-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [25d8195c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:10:02,348-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [25d8195c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:10:02,395-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:10:02,395-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:10:02,395-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:10:02,395-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:10:02,395-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3b7ebb04
>2018-05-30 03:10:02,395-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:10:02,408-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [3e26e1f1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:10:02,409-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3e26e1f1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 614016a6
>2018-05-30 03:10:02,411-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3e26e1f1] FINISH, SetVdsStatusVDSCommand, log id: 614016a6
>2018-05-30 03:10:02,416-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3e26e1f1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3e26e1f1, Job ID: 6628effd-e09f-4072-88bc-dc8cd1a57db9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:10:02,421-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3e26e1f1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2f84a16b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:10:02,423-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3e26e1f1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2f84a16b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:10:02,433-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [45f5e41a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:10:02,434-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [45f5e41a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:15:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3c31af0b] Autorecovering 2 hosts
>2018-05-30 03:15:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3c31af0b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:15:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d6e1b9e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:15:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d6e1b9e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:15:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d6e1b9e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:15:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d6e1b9e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:15:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3d6e1b9e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72487e39
>2018-05-30 03:15:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3d6e1b9e] FINISH, SetVdsStatusVDSCommand, log id: 72487e39
>2018-05-30 03:15:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d6e1b9e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:15:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d6e1b9e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:15:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3d6e1b9e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:15:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6b3e6f4a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:15:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6b3e6f4a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:15:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6b3e6f4a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:15:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6b3e6f4a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:15:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6b3e6f4a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5bcd9ba7
>2018-05-30 03:15:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6b3e6f4a] FINISH, SetVdsStatusVDSCommand, log id: 5bcd9ba7
>2018-05-30 03:15:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [6b3e6f4a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 24c6d7c
>2018-05-30 03:15:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [6b3e6f4a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 24c6d7c
>2018-05-30 03:15:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6b3e6f4a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:15:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6b3e6f4a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:15:02,983-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [21063390] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 17ca6825
>2018-05-30 03:15:02,985-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [21063390] FINISH, GetHardwareInfoVDSCommand, log id: 17ca6825
>2018-05-30 03:15:02,998-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [3fbe9149] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:15:03,007-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [6f2d8818] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:15:03,012-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [3922b1a0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:15:03,023-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [3922b1a0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: b278ae0
>2018-05-30 03:15:03,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [26729fb7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: c563eb3
>2018-05-30 03:15:03,103-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [26729fb7] FINISH, GetHardwareInfoVDSCommand, log id: c563eb3
>2018-05-30 03:15:03,115-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [796ad9d1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:15:03,120-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [2c5d03de] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:15:03,125-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [732f3621] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:15:03,131-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [732f3621] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 376bf3ee
>2018-05-30 03:15:03,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3922b1a0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:15:03,401-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [3922b1a0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: b278ae0
>2018-05-30 03:15:03,405-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3922b1a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:15:03,406-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3922b1a0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:15:03,410-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3922b1a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:15:03,411-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3922b1a0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:15:03,414-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3922b1a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:15:03,415-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3922b1a0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:15:03,418-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [3922b1a0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:15:03,421-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3922b1a0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3922b1a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:15:03,427-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3e35c91
>2018-05-30 03:15:03,427-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:15:03,509-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [732f3621] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:15:03,509-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [732f3621] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 376bf3ee
>2018-05-30 03:15:03,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [732f3621] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:15:03,514-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [732f3621] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:15:03,518-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [732f3621] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:15:03,519-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [732f3621] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:15:03,522-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [732f3621] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:15:03,523-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [732f3621] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:15:03,525-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:15:03,526-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:15:03,526-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:15:03,526-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:15:03,526-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3e35c91
>2018-05-30 03:15:03,526-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:15:03,527-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [732f3621] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:15:03,530-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [732f3621] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 732f3621, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:15:03,538-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6d08a357
>2018-05-30 03:15:03,538-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:15:03,540-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [3373632] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:15:03,542-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3373632] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 513463d1
>2018-05-30 03:15:03,544-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3373632] FINISH, SetVdsStatusVDSCommand, log id: 513463d1
>2018-05-30 03:15:03,576-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3373632] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3373632, Job ID: 685aa4a7-e503-4b43-8621-a8e961d039b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:15:03,581-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3373632] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6f2d8818, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:15:03,583-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3373632] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6f2d8818, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:15:03,594-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [130ead5d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:15:03,595-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [130ead5d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:15:03,640-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:15:03,640-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:15:03,640-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:15:03,640-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:15:03,640-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6d08a357
>2018-05-30 03:15:03,640-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:15:03,653-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [383c6363] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:15:03,654-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [383c6363] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1cb3297
>2018-05-30 03:15:03,656-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [383c6363] FINISH, SetVdsStatusVDSCommand, log id: 1cb3297
>2018-05-30 03:15:03,662-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [383c6363] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 383c6363, Job ID: 1b401707-df04-4f52-81b9-9eea1001a99b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:15:03,666-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [383c6363] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2c5d03de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:15:03,669-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [383c6363] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2c5d03de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:15:03,679-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [6774119a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:15:03,680-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [6774119a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7ff116c1] Autorecovering 2 hosts
>2018-05-30 03:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7ff116c1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:20:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51610388] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51610388] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51610388] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51610388] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:20:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [51610388] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 425848a7
>2018-05-30 03:20:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [51610388] FINISH, SetVdsStatusVDSCommand, log id: 425848a7
>2018-05-30 03:20:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51610388] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:20:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51610388] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:20:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [51610388] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:20:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [66d089de] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:20:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [66d089de] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:20:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [66d089de] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:20:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [66d089de] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:20:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [66d089de] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1c0717f2
>2018-05-30 03:20:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [66d089de] FINISH, SetVdsStatusVDSCommand, log id: 1c0717f2
>2018-05-30 03:20:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [66d089de] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 308daf2b
>2018-05-30 03:20:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [66d089de] FINISH, SetHaMaintenanceModeVDSCommand, log id: 308daf2b
>2018-05-30 03:20:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [66d089de] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:20:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [66d089de] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:20:01,226-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4a5ef427] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3136387b
>2018-05-30 03:20:01,228-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4a5ef427] FINISH, GetHardwareInfoVDSCommand, log id: 3136387b
>2018-05-30 03:20:01,241-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7aaaec3a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:20:01,250-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [52eea52e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:20:01,255-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [7423662] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:20:01,267-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [7423662] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e82b976
>2018-05-30 03:20:01,341-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [1ccc1792] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 735749d1
>2018-05-30 03:20:01,343-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [1ccc1792] FINISH, GetHardwareInfoVDSCommand, log id: 735749d1
>2018-05-30 03:20:01,356-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [3a51f52d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:20:01,360-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [c51ce3e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:20:01,365-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2d9c22fc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:20:01,371-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2d9c22fc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 685d32c9
>2018-05-30 03:20:01,653-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7423662] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:20:01,653-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [7423662] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e82b976
>2018-05-30 03:20:01,657-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7423662] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:20:01,658-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7423662] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:20:01,662-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7423662] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:20:01,663-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7423662] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:20:01,666-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7423662] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:20:01,667-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7423662] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:20:01,670-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [7423662] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:20:01,673-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7423662] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7423662, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:20:01,676-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2b95fef2
>2018-05-30 03:20:01,676-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:20:01,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:20:01,773-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:20:01,773-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:20:01,773-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:20:01,773-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2b95fef2
>2018-05-30 03:20:01,773-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:20:01,787-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [5c0184b0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:20:01,788-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5c0184b0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47aa3393
>2018-05-30 03:20:01,790-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5c0184b0] FINISH, SetVdsStatusVDSCommand, log id: 47aa3393
>2018-05-30 03:20:01,821-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5c0184b0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5c0184b0, Job ID: f7ab1000-0e8b-4ad8-ad1c-e380e047d87b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:20:01,826-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5c0184b0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 52eea52e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:20:01,828-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5c0184b0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 52eea52e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:20:01,839-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [3ceca5f8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:20:01,840-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [3ceca5f8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:20:02,735-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2d9c22fc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:20:02,735-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2d9c22fc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 685d32c9
>2018-05-30 03:20:02,739-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2d9c22fc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:20:02,740-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2d9c22fc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:20:02,744-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2d9c22fc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:20:02,744-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2d9c22fc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:20:02,748-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2d9c22fc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:20:02,749-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2d9c22fc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:20:02,752-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2d9c22fc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:20:02,755-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2d9c22fc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2d9c22fc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:20:02,758-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 611aa3d9
>2018-05-30 03:20:02,758-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:20:02,854-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:20:02,854-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:20:02,854-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:20:02,854-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:20:02,854-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 611aa3d9
>2018-05-30 03:20:02,854-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:20:02,868-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [96645e6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:20:02,869-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [96645e6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 16dbb2f8
>2018-05-30 03:20:02,871-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [96645e6] FINISH, SetVdsStatusVDSCommand, log id: 16dbb2f8
>2018-05-30 03:20:02,877-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [96645e6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 96645e6, Job ID: 87307ee1-474f-4ddc-a4a1-cffeb87828bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:20:02,882-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [96645e6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: c51ce3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:20:02,884-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [96645e6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: c51ce3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:20:02,894-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [72c9c024] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:20:02,895-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [72c9c024] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:25:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [10bd92a2] Autorecovering 2 hosts
>2018-05-30 03:25:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [10bd92a2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:25:00,035-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3a978923] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:25:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3a978923] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:25:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3a978923] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:25:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3a978923] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:25:00,045-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3a978923] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1e16731b
>2018-05-30 03:25:00,047-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3a978923] FINISH, SetVdsStatusVDSCommand, log id: 1e16731b
>2018-05-30 03:25:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3a978923] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:25:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3a978923] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:25:00,050-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3a978923] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:25:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [296e4021] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:25:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [296e4021] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:25:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [296e4021] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:25:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [296e4021] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:25:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [296e4021] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76a14a5f
>2018-05-30 03:25:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [296e4021] FINISH, SetVdsStatusVDSCommand, log id: 76a14a5f
>2018-05-30 03:25:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [296e4021] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1682ff99
>2018-05-30 03:25:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [296e4021] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1682ff99
>2018-05-30 03:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [296e4021] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:25:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [296e4021] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:25:00,563-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [5dd08b13] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 42979ad9
>2018-05-30 03:25:00,564-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [5dd08b13] FINISH, GetHardwareInfoVDSCommand, log id: 42979ad9
>2018-05-30 03:25:00,580-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [7d208450] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:25:00,584-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [33df88a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:25:00,590-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [49153462] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:25:00,604-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [49153462] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e05a186
>2018-05-30 03:25:01,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49153462] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:25:01,983-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [49153462] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e05a186
>2018-05-30 03:25:01,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49153462] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:25:01,988-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [49153462] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:25:01,992-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49153462] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:25:01,992-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [49153462] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:25:01,996-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49153462] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:25:01,997-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [49153462] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:25:02,001-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [49153462] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:25:02,004-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49153462] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 49153462, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:25:02,011-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 15300b65
>2018-05-30 03:25:02,011-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:25:02,107-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:25:02,107-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:25:02,107-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:25:02,107-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:25:02,107-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 15300b65
>2018-05-30 03:25:02,107-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:25:02,121-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [7898808b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:25:02,122-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7898808b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b91caf1
>2018-05-30 03:25:02,124-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7898808b] FINISH, SetVdsStatusVDSCommand, log id: b91caf1
>2018-05-30 03:25:02,133-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7898808b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7898808b, Job ID: c41b7355-e751-4ee8-a161-4d584b8b8272, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:25:02,138-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7898808b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 33df88a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:25:02,140-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7898808b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 33df88a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:25:02,152-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [3153b12] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:25:02,153-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [3153b12] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:25:02,438-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [666e3835] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7f30eeea
>2018-05-30 03:25:02,440-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [666e3835] FINISH, GetHardwareInfoVDSCommand, log id: 7f30eeea
>2018-05-30 03:25:02,452-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [18624a50] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:25:02,457-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [36d848fe] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:25:02,462-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [42012828] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:25:02,469-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [42012828] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 100244c7
>2018-05-30 03:25:02,845-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [42012828] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:25:02,845-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [42012828] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 100244c7
>2018-05-30 03:25:02,849-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [42012828] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:25:02,849-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [42012828] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:25:02,854-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [42012828] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:25:02,854-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [42012828] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:25:02,858-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [42012828] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:25:02,859-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [42012828] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:25:02,862-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [42012828] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:25:02,864-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [42012828] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 42012828, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:25:02,867-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 78634e16
>2018-05-30 03:25:02,867-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:25:02,963-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:25:02,963-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:25:02,963-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:25:02,964-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:25:02,964-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 78634e16
>2018-05-30 03:25:02,964-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:25:02,977-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [a753877] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:25:02,978-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [a753877] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 574df750
>2018-05-30 03:25:02,980-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [a753877] FINISH, SetVdsStatusVDSCommand, log id: 574df750
>2018-05-30 03:25:02,986-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [a753877] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: a753877, Job ID: d39ecf41-f72e-4e87-afc8-35c08536e2f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:25:02,991-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [a753877] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 36d848fe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:25:02,993-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [a753877] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 36d848fe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:25:03,005-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [6d93c232] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:25:03,007-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [6d93c232] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [643f9577] Autorecovering 2 hosts
>2018-05-30 03:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [643f9577] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:30:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8305d8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:30:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8305d8] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:30:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8305d8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:30:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8305d8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:30:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3a8305d8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33854b5f
>2018-05-30 03:30:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3a8305d8] FINISH, SetVdsStatusVDSCommand, log id: 33854b5f
>2018-05-30 03:30:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8305d8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:30:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8305d8] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:30:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3a8305d8] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:30:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [34f232d4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:30:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [34f232d4] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:30:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [34f232d4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:30:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [34f232d4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:30:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [34f232d4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: e19fb89
>2018-05-30 03:30:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [34f232d4] FINISH, SetVdsStatusVDSCommand, log id: e19fb89
>2018-05-30 03:30:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [34f232d4] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: bbad2ea
>2018-05-30 03:30:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [34f232d4] FINISH, SetHaMaintenanceModeVDSCommand, log id: bbad2ea
>2018-05-30 03:30:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [34f232d4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:30:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [34f232d4] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:30:00,613-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6fc7f2e4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5e93b5a1
>2018-05-30 03:30:00,615-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6fc7f2e4] FINISH, GetHardwareInfoVDSCommand, log id: 5e93b5a1
>2018-05-30 03:30:00,629-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [1641f891] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:30:00,638-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [f35f845] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:30:00,643-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [348911f7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:30:00,656-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [348911f7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 157566bc
>2018-05-30 03:30:01,038-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [348911f7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:30:01,038-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [348911f7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 157566bc
>2018-05-30 03:30:01,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [348911f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:30:01,043-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [348911f7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:30:01,046-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [348911f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:30:01,047-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [348911f7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:30:01,051-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [348911f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:30:01,052-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [348911f7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:30:01,054-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [348911f7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:30:01,057-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [348911f7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 348911f7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:30:01,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 53e374fe
>2018-05-30 03:30:01,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:30:01,158-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:30:01,158-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:30:01,158-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:30:01,158-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:30:01,158-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 53e374fe
>2018-05-30 03:30:01,158-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:30:01,172-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [74de7a89] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:30:01,173-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [74de7a89] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 460e046f
>2018-05-30 03:30:01,175-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [74de7a89] FINISH, SetVdsStatusVDSCommand, log id: 460e046f
>2018-05-30 03:30:01,206-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [74de7a89] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 74de7a89, Job ID: 7a417dac-8e4d-45e9-8843-0870c38bbe01, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:30:01,211-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [74de7a89] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: f35f845, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:30:01,213-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [74de7a89] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: f35f845, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:30:01,225-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [4fcdcb4b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:30:01,226-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [4fcdcb4b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:30:03,680-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [276ff507] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: b7539e3
>2018-05-30 03:30:03,681-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [276ff507] FINISH, GetHardwareInfoVDSCommand, log id: b7539e3
>2018-05-30 03:30:03,696-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [1de008aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:30:03,700-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [27690b83] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:30:03,705-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [35d96649] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:30:03,712-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [35d96649] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 73a677ab
>2018-05-30 03:30:04,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [35d96649] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:30:04,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [35d96649] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 73a677ab
>2018-05-30 03:30:04,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [35d96649] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:30:04,082-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [35d96649] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:30:04,086-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [35d96649] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:30:04,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [35d96649] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:30:04,090-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [35d96649] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:30:04,091-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [35d96649] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:30:04,094-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [35d96649] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:30:04,096-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [35d96649] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 35d96649, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:30:04,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2c32c61f
>2018-05-30 03:30:04,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:30:04,197-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:30:04,197-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:30:04,197-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:30:04,197-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:30:04,197-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2c32c61f
>2018-05-30 03:30:04,197-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:30:04,210-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [1adcc379] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:30:04,211-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1adcc379] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31033401
>2018-05-30 03:30:04,213-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1adcc379] FINISH, SetVdsStatusVDSCommand, log id: 31033401
>2018-05-30 03:30:04,219-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1adcc379] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1adcc379, Job ID: 7a32be2b-3a80-4f88-bbda-0a4a4776294d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:30:04,223-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1adcc379] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 27690b83, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:30:04,226-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1adcc379] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 27690b83, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:30:04,236-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [7c2517b1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:30:04,237-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [7c2517b1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:35:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [68ab1ecd] Autorecovering 2 hosts
>2018-05-30 03:35:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [68ab1ecd] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:35:00,030-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1f4f9064] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:35:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1f4f9064] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:35:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1f4f9064] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:35:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1f4f9064] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:35:00,039-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1f4f9064] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7630768
>2018-05-30 03:35:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1f4f9064] FINISH, SetVdsStatusVDSCommand, log id: 7630768
>2018-05-30 03:35:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1f4f9064] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:35:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1f4f9064] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:35:00,045-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1f4f9064] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:35:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1dbb105a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:35:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1dbb105a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:35:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1dbb105a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:35:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1dbb105a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:35:00,058-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1dbb105a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1fec2b55
>2018-05-30 03:35:00,060-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1dbb105a] FINISH, SetVdsStatusVDSCommand, log id: 1fec2b55
>2018-05-30 03:35:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [1dbb105a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4dc04ba8
>2018-05-30 03:35:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [1dbb105a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4dc04ba8
>2018-05-30 03:35:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1dbb105a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:35:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1dbb105a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:35:01,880-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [15e15557] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1f333b94
>2018-05-30 03:35:01,881-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [15e15557] FINISH, GetHardwareInfoVDSCommand, log id: 1f333b94
>2018-05-30 03:35:01,894-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [504880c5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:35:01,903-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [198eab37] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:35:01,908-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2822fc33] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:35:01,920-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2822fc33] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3af10a87
>2018-05-30 03:35:02,133-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [a774e28] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4d3777a4
>2018-05-30 03:35:02,135-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [a774e28] FINISH, GetHardwareInfoVDSCommand, log id: 4d3777a4
>2018-05-30 03:35:02,148-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [1a472ac0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:35:02,153-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [542a8ace] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:35:02,158-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2037b595] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:35:02,164-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2037b595] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1a946ef8
>2018-05-30 03:35:02,541-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2037b595] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:35:02,541-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2037b595] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1a946ef8
>2018-05-30 03:35:02,545-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2037b595] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:35:02,545-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2037b595] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:35:02,549-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2037b595] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:35:02,550-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2037b595] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:35:02,554-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2037b595] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:35:02,555-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2037b595] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:35:02,557-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2037b595] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:35:02,560-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2037b595] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2037b595, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:35:02,566-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4638cf9b
>2018-05-30 03:35:02,566-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:35:02,663-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:35:02,663-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:35:02,663-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:35:02,663-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:35:02,663-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4638cf9b
>2018-05-30 03:35:02,663-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:35:02,677-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [18ae67d4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:35:02,678-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [18ae67d4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 73ce5622
>2018-05-30 03:35:02,681-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [18ae67d4] FINISH, SetVdsStatusVDSCommand, log id: 73ce5622
>2018-05-30 03:35:02,715-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ae67d4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 18ae67d4, Job ID: b80cf50d-f814-4538-93f7-152a34202676, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:35:02,720-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ae67d4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 542a8ace, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:35:02,722-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ae67d4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 542a8ace, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:35:02,732-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [6892754e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:35:02,733-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [6892754e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:35:03,239-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2822fc33] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:35:03,239-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2822fc33] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3af10a87
>2018-05-30 03:35:03,242-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2822fc33] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:35:03,243-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2822fc33] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:35:03,247-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2822fc33] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:35:03,247-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2822fc33] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:35:03,251-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2822fc33] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:35:03,252-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2822fc33] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:35:03,255-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2822fc33] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:35:03,257-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2822fc33] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2822fc33, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:35:03,260-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 19dd29c8
>2018-05-30 03:35:03,260-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:35:03,360-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:35:03,360-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:35:03,360-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:35:03,360-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:35:03,360-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 19dd29c8
>2018-05-30 03:35:03,360-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:35:03,372-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [11999eab] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:35:03,373-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [11999eab] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d790547
>2018-05-30 03:35:03,375-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [11999eab] FINISH, SetVdsStatusVDSCommand, log id: 2d790547
>2018-05-30 03:35:03,380-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [11999eab] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 11999eab, Job ID: 665f4d45-f633-4e6e-9da8-4225a657efad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:35:03,385-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [11999eab] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 198eab37, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:35:03,387-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [11999eab] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 198eab37, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:35:03,400-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [4feadf50] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:35:03,401-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [4feadf50] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:35:35,000-04 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler8) [78846500] Start deleteAgedOutAuditLogs
>2018-05-30 03:35:35,001-04 INFO  [org.ovirt.engine.core.bll.CommandEntityCleanupManager] (DefaultQuartzScheduler6) [41d37a84] Start deleteAgedOutCommandEntities
>2018-05-30 03:35:35,001-04 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler8) [78846500] Finished deleteAgedOutAuditLogs
>2018-05-30 03:35:35,001-04 INFO  [org.ovirt.engine.core.bll.CommandEntityCleanupManager] (DefaultQuartzScheduler6) [41d37a84] Finished deleteAgedOutCommandEntities
>2018-05-30 03:40:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5a0f002c] Autorecovering 2 hosts
>2018-05-30 03:40:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5a0f002c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:40:00,032-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [12cdc016] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:40:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [12cdc016] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:40:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [12cdc016] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:40:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [12cdc016] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:40:00,040-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [12cdc016] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4f69088a
>2018-05-30 03:40:00,043-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [12cdc016] FINISH, SetVdsStatusVDSCommand, log id: 4f69088a
>2018-05-30 03:40:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [12cdc016] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:40:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [12cdc016] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:40:00,046-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [12cdc016] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:40:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4523baa6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:40:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4523baa6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:40:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4523baa6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:40:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4523baa6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:40:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4523baa6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50c8c9fb
>2018-05-30 03:40:00,081-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4523baa6] FINISH, SetVdsStatusVDSCommand, log id: 50c8c9fb
>2018-05-30 03:40:00,084-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4523baa6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5078870
>2018-05-30 03:40:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4523baa6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5078870
>2018-05-30 03:40:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4523baa6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:40:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4523baa6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:40:00,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4ec06d12] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1ea2586f
>2018-05-30 03:40:00,282-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4ec06d12] FINISH, GetHardwareInfoVDSCommand, log id: 1ea2586f
>2018-05-30 03:40:00,294-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [42a69544] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:40:00,299-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [29746c75] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:40:00,304-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [9f865] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:40:00,315-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [9f865] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6682046b
>2018-05-30 03:40:00,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [9f865] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:40:00,696-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [9f865] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6682046b
>2018-05-30 03:40:00,699-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [9f865] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:40:00,700-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [9f865] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:40:00,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [9f865] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:40:00,705-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [9f865] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:40:00,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [9f865] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:40:00,709-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [9f865] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:40:00,712-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [9f865] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:40:00,714-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [9f865] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 9f865, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:40:00,721-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 584959c9
>2018-05-30 03:40:00,721-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:40:00,818-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:40:00,818-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:40:00,818-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:40:00,818-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:40:00,818-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 584959c9
>2018-05-30 03:40:00,818-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:40:00,831-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4e4cd684] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:40:00,833-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4e4cd684] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7dd36fb2
>2018-05-30 03:40:00,835-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4e4cd684] FINISH, SetVdsStatusVDSCommand, log id: 7dd36fb2
>2018-05-30 03:40:00,842-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e4cd684] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4e4cd684, Job ID: 1ea7b0da-4d80-4d83-9b25-5e0ba4667343, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:40:00,847-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e4cd684] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29746c75, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:40:00,850-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e4cd684] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29746c75, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:40:00,860-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [6cad0f8c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:40:00,861-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [6cad0f8c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:40:01,021-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6a584850] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 13d69abf
>2018-05-30 03:40:01,023-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6a584850] FINISH, GetHardwareInfoVDSCommand, log id: 13d69abf
>2018-05-30 03:40:01,035-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [7613fe22] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:40:01,039-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [40d72ee0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:40:01,044-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [75fd1f76] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:40:01,051-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [75fd1f76] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 576c55a7
>2018-05-30 03:40:02,442-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [75fd1f76] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:40:02,442-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [75fd1f76] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 576c55a7
>2018-05-30 03:40:02,446-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [75fd1f76] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:40:02,447-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [75fd1f76] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:40:02,450-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [75fd1f76] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:40:02,451-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [75fd1f76] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:40:02,455-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [75fd1f76] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:40:02,456-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [75fd1f76] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:40:02,459-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [75fd1f76] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:40:02,461-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [75fd1f76] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 75fd1f76, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:40:02,464-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6f106980
>2018-05-30 03:40:02,464-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:40:02,562-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:40:02,562-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:40:02,562-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:40:02,562-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:40:02,562-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6f106980
>2018-05-30 03:40:02,562-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:40:02,575-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [42b4ec83] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:40:02,576-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [42b4ec83] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bbc9e51
>2018-05-30 03:40:02,577-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [42b4ec83] FINISH, SetVdsStatusVDSCommand, log id: 3bbc9e51
>2018-05-30 03:40:02,583-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42b4ec83] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 42b4ec83, Job ID: 9b517413-fd5e-41bd-9a1e-8ed6c64432e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:40:02,587-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42b4ec83] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 40d72ee0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:40:02,589-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42b4ec83] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 40d72ee0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:40:02,600-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [52553853] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:40:02,601-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [52553853] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [4e42bd9a] Autorecovering 2 hosts
>2018-05-30 03:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [4e42bd9a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:45:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1e1b6226] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1e1b6226] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1e1b6226] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1e1b6226] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:45:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1e1b6226] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49aeabd5
>2018-05-30 03:45:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1e1b6226] FINISH, SetVdsStatusVDSCommand, log id: 49aeabd5
>2018-05-30 03:45:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1e1b6226] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:45:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1e1b6226] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:45:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [1e1b6226] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:45:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [da56f4a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [da56f4a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [da56f4a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [da56f4a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:45:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [da56f4a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 40b231a
>2018-05-30 03:45:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [da56f4a] FINISH, SetVdsStatusVDSCommand, log id: 40b231a
>2018-05-30 03:45:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [da56f4a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2e5e50f
>2018-05-30 03:45:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [da56f4a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2e5e50f
>2018-05-30 03:45:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [da56f4a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:45:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [da56f4a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:45:00,349-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3192d626] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5e911f4d
>2018-05-30 03:45:00,351-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3192d626] FINISH, GetHardwareInfoVDSCommand, log id: 5e911f4d
>2018-05-30 03:45:00,364-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3026b0db] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:45:00,372-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [7081c77e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:45:00,377-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6b93b87b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:45:00,388-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6b93b87b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 48262122
>2018-05-30 03:45:00,763-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b93b87b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:45:00,763-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6b93b87b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 48262122
>2018-05-30 03:45:00,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b93b87b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:45:00,768-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6b93b87b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:45:00,772-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b93b87b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:45:00,772-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6b93b87b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:45:00,776-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b93b87b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:45:00,777-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6b93b87b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:45:00,779-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6b93b87b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:45:00,782-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b93b87b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6b93b87b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:45:00,789-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6a7e0389
>2018-05-30 03:45:00,789-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:45:00,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:45:00,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:45:00,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:45:00,886-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:45:00,886-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6a7e0389
>2018-05-30 03:45:00,886-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:45:00,898-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [1ba1cbf5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:45:00,900-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1ba1cbf5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b31837e
>2018-05-30 03:45:00,901-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1ba1cbf5] FINISH, SetVdsStatusVDSCommand, log id: 3b31837e
>2018-05-30 03:45:00,934-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1ba1cbf5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1ba1cbf5, Job ID: 10da1865-ef13-4dab-b102-0dfa0d48a1ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:45:00,938-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1ba1cbf5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7081c77e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:45:00,940-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1ba1cbf5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7081c77e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:45:00,951-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [728cd0bf] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:45:00,952-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [728cd0bf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:45:01,708-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [48d96b86] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 334801d2
>2018-05-30 03:45:01,709-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [48d96b86] FINISH, GetHardwareInfoVDSCommand, log id: 334801d2
>2018-05-30 03:45:01,721-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [2c54547e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:45:01,725-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [33c4bd1c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:45:01,730-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [59b3b2aa] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:45:01,737-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [59b3b2aa] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3f1ff772
>2018-05-30 03:45:02,112-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59b3b2aa] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:45:02,112-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [59b3b2aa] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3f1ff772
>2018-05-30 03:45:02,116-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59b3b2aa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:45:02,117-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [59b3b2aa] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:45:02,120-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59b3b2aa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:45:02,121-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [59b3b2aa] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:45:02,125-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59b3b2aa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:45:02,126-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [59b3b2aa] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:45:02,129-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [59b3b2aa] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:45:02,132-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59b3b2aa] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 59b3b2aa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:45:02,135-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3afabf40
>2018-05-30 03:45:02,135-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:45:02,229-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:45:02,229-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:45:02,229-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:45:02,229-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:45:02,229-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3afabf40
>2018-05-30 03:45:02,229-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:45:02,241-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [6908a261] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:45:02,242-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6908a261] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52331419
>2018-05-30 03:45:02,244-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6908a261] FINISH, SetVdsStatusVDSCommand, log id: 52331419
>2018-05-30 03:45:02,250-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6908a261] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6908a261, Job ID: 8068a67e-4f0f-434f-b9d8-394fecbb3ed8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:45:02,254-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6908a261] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 33c4bd1c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:45:02,256-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6908a261] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 33c4bd1c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:45:02,267-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [71a365a0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:45:02,268-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [71a365a0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [315ad53f] Autorecovering 2 hosts
>2018-05-30 03:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [315ad53f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:50:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [36c189f0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:50:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [36c189f0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:50:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [36c189f0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:50:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [36c189f0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:50:00,055-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [36c189f0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6eb00a9d
>2018-05-30 03:50:00,058-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [36c189f0] FINISH, SetVdsStatusVDSCommand, log id: 6eb00a9d
>2018-05-30 03:50:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [36c189f0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:50:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [36c189f0] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:50:00,061-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [36c189f0] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:50:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [500ad926] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:50:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [500ad926] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:50:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [500ad926] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:50:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [500ad926] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:50:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [500ad926] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18829dac
>2018-05-30 03:50:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [500ad926] FINISH, SetVdsStatusVDSCommand, log id: 18829dac
>2018-05-30 03:50:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [500ad926] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1449ab7d
>2018-05-30 03:50:00,082-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [500ad926] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1449ab7d
>2018-05-30 03:50:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [500ad926] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:50:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [500ad926] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:50:01,630-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7a075a3c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 599197fc
>2018-05-30 03:50:01,632-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7a075a3c] FINISH, GetHardwareInfoVDSCommand, log id: 599197fc
>2018-05-30 03:50:01,645-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [68045b3b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:50:01,653-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [dc41994] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:50:01,658-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [69d32300] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:50:01,717-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [69d32300] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f906afb
>2018-05-30 03:50:02,916-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7d845e58] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 38095985
>2018-05-30 03:50:02,918-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7d845e58] FINISH, GetHardwareInfoVDSCommand, log id: 38095985
>2018-05-30 03:50:02,930-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [464524f6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:50:02,935-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [21f7ca0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:50:02,939-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [77a026f7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:50:02,946-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [77a026f7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1257571c
>2018-05-30 03:50:03,090-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d32300] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:50:03,090-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [69d32300] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f906afb
>2018-05-30 03:50:03,094-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d32300] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:50:03,094-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [69d32300] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:50:03,098-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d32300] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:50:03,099-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [69d32300] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:50:03,102-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d32300] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:50:03,103-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [69d32300] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:50:03,106-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [69d32300] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:50:03,108-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d32300] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 69d32300, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:50:03,115-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2231d526
>2018-05-30 03:50:03,115-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:50:03,213-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:50:03,213-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:50:03,213-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:50:03,213-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:50:03,213-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2231d526
>2018-05-30 03:50:03,213-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:50:03,225-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [7f0fcfb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:50:03,227-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7f0fcfb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 24fe278f
>2018-05-30 03:50:03,229-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7f0fcfb] FINISH, SetVdsStatusVDSCommand, log id: 24fe278f
>2018-05-30 03:50:03,260-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f0fcfb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7f0fcfb, Job ID: 2fe1d799-0df1-4743-a426-f025df17bcf5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:50:03,264-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f0fcfb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: dc41994, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:50:03,267-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f0fcfb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: dc41994, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:50:03,277-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [76c2bed7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:50:03,278-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [76c2bed7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:50:03,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77a026f7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:50:03,332-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [77a026f7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1257571c
>2018-05-30 03:50:03,336-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77a026f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:50:03,337-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [77a026f7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:50:03,341-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77a026f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:50:03,341-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [77a026f7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:50:03,345-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77a026f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:50:03,346-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [77a026f7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:50:03,349-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [77a026f7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:50:03,351-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77a026f7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 77a026f7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:50:03,354-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: f9069c3
>2018-05-30 03:50:03,354-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:50:03,453-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:50:03,453-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:50:03,453-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:50:03,453-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:50:03,453-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: f9069c3
>2018-05-30 03:50:03,453-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:50:03,466-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [46ecfa1d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:50:03,467-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [46ecfa1d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26b28e78
>2018-05-30 03:50:03,469-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [46ecfa1d] FINISH, SetVdsStatusVDSCommand, log id: 26b28e78
>2018-05-30 03:50:03,474-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [46ecfa1d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 46ecfa1d, Job ID: 24910893-524c-4585-a6bc-6c97c321994d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:50:03,479-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [46ecfa1d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 21f7ca0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:50:03,481-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [46ecfa1d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 21f7ca0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:50:03,491-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [59a6eea6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:50:03,492-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [59a6eea6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:54:43,484-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler9) [3123f807] Attempting to update VMs/Templates Ovf.
>2018-05-30 03:54:43,487-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 03:54:43,487-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 03:54:43,487-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:54:43,539-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-30 03:54:43,543-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-30 03:54:43,543-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Attempting to update template OVFs in Data Center 'Default'
>2018-05-30 03:54:43,543-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-30 03:54:43,543-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 03:54:43,543-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 03:54:43,544-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 03:55:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [81e6294] Autorecovering 2 hosts
>2018-05-30 03:55:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [81e6294] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:55:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [d0abe31] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:55:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [d0abe31] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:55:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [d0abe31] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:55:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [d0abe31] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:55:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [d0abe31] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6c8ecdc5
>2018-05-30 03:55:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [d0abe31] FINISH, SetVdsStatusVDSCommand, log id: 6c8ecdc5
>2018-05-30 03:55:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [d0abe31] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:55:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [d0abe31] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 03:55:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [d0abe31] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:55:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [627a864b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:55:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [627a864b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 03:55:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [627a864b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:55:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [627a864b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:55:00,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [627a864b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 222e330a
>2018-05-30 03:55:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [627a864b] FINISH, SetVdsStatusVDSCommand, log id: 222e330a
>2018-05-30 03:55:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [627a864b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3c6bed25
>2018-05-30 03:55:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [627a864b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3c6bed25
>2018-05-30 03:55:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [627a864b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 03:55:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [627a864b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 03:55:00,986-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3aa26d3e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 179bf5a2
>2018-05-30 03:55:00,988-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3aa26d3e] FINISH, GetHardwareInfoVDSCommand, log id: 179bf5a2
>2018-05-30 03:55:01,000-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [151c78e6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:55:01,009-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [6738519e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:55:01,014-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [15a84530] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:55:01,025-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [15a84530] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1370a022
>2018-05-30 03:55:01,062-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2794b0b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 24479e05
>2018-05-30 03:55:01,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2794b0b] FINISH, GetHardwareInfoVDSCommand, log id: 24479e05
>2018-05-30 03:55:01,076-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [675cd75e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:55:01,080-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [342d162b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:55:01,085-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [976f5d3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 03:55:01,091-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [976f5d3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 740ec081
>2018-05-30 03:55:01,395-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [15a84530] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:55:01,395-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [15a84530] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1370a022
>2018-05-30 03:55:01,399-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [15a84530] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:55:01,400-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [15a84530] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:55:01,404-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [15a84530] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:55:01,404-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [15a84530] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:55:01,408-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [15a84530] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:55:01,409-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [15a84530] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:55:01,412-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [15a84530] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:55:01,415-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [15a84530] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 15a84530, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:55:01,421-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 17bd5af9
>2018-05-30 03:55:01,422-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:55:01,462-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [976f5d3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 03:55:01,462-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [976f5d3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 740ec081
>2018-05-30 03:55:01,465-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [976f5d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:55:01,466-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [976f5d3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:55:01,470-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [976f5d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:55:01,471-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [976f5d3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:55:01,475-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [976f5d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 03:55:01,476-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [976f5d3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 03:55:01,479-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [976f5d3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 03:55:01,481-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [976f5d3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 976f5d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 03:55:01,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:55:01,521-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:55:01,521-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:55:01,521-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:55:01,521-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 17bd5af9
>2018-05-30 03:55:01,521-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:55:01,523-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7e713d86
>2018-05-30 03:55:01,523-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 03:55:01,535-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [14f71e8d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:55:01,536-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [14f71e8d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 29822f1d
>2018-05-30 03:55:01,538-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [14f71e8d] FINISH, SetVdsStatusVDSCommand, log id: 29822f1d
>2018-05-30 03:55:01,569-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [14f71e8d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 14f71e8d, Job ID: 33f11979-0beb-4bd1-9066-f814ca0cb9e6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:55:01,573-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [14f71e8d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6738519e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:55:01,576-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [14f71e8d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6738519e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:55:01,586-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [2ec9ecd] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 03:55:01,587-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [2ec9ecd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 03:55:01,623-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:55:01,623-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 03:55:01,623-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 03:55:01,623-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 03:55:01,623-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7e713d86
>2018-05-30 03:55:01,623-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 03:55:01,636-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5f75cc1c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:55:01,637-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5f75cc1c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6406dae0
>2018-05-30 03:55:01,639-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5f75cc1c] FINISH, SetVdsStatusVDSCommand, log id: 6406dae0
>2018-05-30 03:55:01,645-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5f75cc1c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5f75cc1c, Job ID: ece4c200-8188-469c-a6f8-bc8a88d9c8fe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 03:55:01,649-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5f75cc1c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 342d162b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 03:55:01,651-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5f75cc1c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 342d162b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 03:55:01,662-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [8f2580d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 03:55:01,663-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [8f2580d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7f2a0cdb] Autorecovering 2 hosts
>2018-05-30 04:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7f2a0cdb] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:00:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1a57c8a8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:00:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1a57c8a8] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:00:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1a57c8a8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:00:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1a57c8a8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:00:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1a57c8a8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26257a80
>2018-05-30 04:00:00,079-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1a57c8a8] FINISH, SetVdsStatusVDSCommand, log id: 26257a80
>2018-05-30 04:00:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1a57c8a8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:00:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1a57c8a8] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:00:00,082-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1a57c8a8] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:00:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [32be9a51] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:00:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [32be9a51] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:00:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [32be9a51] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:00:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [32be9a51] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:00:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [32be9a51] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 769386b0
>2018-05-30 04:00:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [32be9a51] FINISH, SetVdsStatusVDSCommand, log id: 769386b0
>2018-05-30 04:00:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [32be9a51] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 697f96e9
>2018-05-30 04:00:00,107-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [32be9a51] FINISH, SetHaMaintenanceModeVDSCommand, log id: 697f96e9
>2018-05-30 04:00:00,107-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [32be9a51] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:00:00,108-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [32be9a51] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:00:02,520-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [16daab60] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 112fba8
>2018-05-30 04:00:02,522-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [16daab60] FINISH, GetHardwareInfoVDSCommand, log id: 112fba8
>2018-05-30 04:00:02,536-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3f917b84] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:00:02,544-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [6f98bad7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:00:02,549-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [672e3a48] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:00:02,561-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [672e3a48] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2389a7d2
>2018-05-30 04:00:02,571-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [48c1f028] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 57eec0d2
>2018-05-30 04:00:02,573-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [48c1f028] FINISH, GetHardwareInfoVDSCommand, log id: 57eec0d2
>2018-05-30 04:00:02,585-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [142b30a6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:00:02,590-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [33a2973b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:00:02,594-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2706b3a0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:00:02,600-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2706b3a0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 698a28c0
>2018-05-30 04:00:02,930-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [672e3a48] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:00:02,930-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [672e3a48] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2389a7d2
>2018-05-30 04:00:02,933-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [672e3a48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:00:02,934-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [672e3a48] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:00:02,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [672e3a48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:00:02,939-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [672e3a48] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:00:02,942-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [672e3a48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:00:02,943-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [672e3a48] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:00:02,946-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [672e3a48] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:00:02,948-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [672e3a48] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 672e3a48, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:00:02,954-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2d29d792
>2018-05-30 04:00:02,954-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:00:02,974-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2706b3a0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:00:02,974-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2706b3a0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 698a28c0
>2018-05-30 04:00:02,977-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2706b3a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:00:02,978-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2706b3a0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:00:02,982-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2706b3a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:00:02,982-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2706b3a0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:00:02,986-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2706b3a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:00:02,987-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2706b3a0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:00:02,989-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2706b3a0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:00:02,992-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2706b3a0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2706b3a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:00:03,054-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:00:03,054-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:00:03,054-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:00:03,054-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:00:03,054-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2d29d792
>2018-05-30 04:00:03,054-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:00:03,056-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 28dfb1f0
>2018-05-30 04:00:03,056-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:00:03,068-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [57b6a322] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:00:03,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [57b6a322] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ad88ba1
>2018-05-30 04:00:03,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [57b6a322] FINISH, SetVdsStatusVDSCommand, log id: ad88ba1
>2018-05-30 04:00:03,103-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b6a322] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 57b6a322, Job ID: 4df62597-0245-4645-a23e-629d603428d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:00:03,107-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b6a322] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6f98bad7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:00:03,109-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b6a322] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6f98bad7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:00:03,120-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [449c40b4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:00:03,121-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [449c40b4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:00:03,155-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:00:03,155-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:00:03,155-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:00:03,155-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:00:03,155-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 28dfb1f0
>2018-05-30 04:00:03,155-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:00:03,168-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [6444e040] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:00:03,169-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6444e040] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3049d267
>2018-05-30 04:00:03,171-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6444e040] FINISH, SetVdsStatusVDSCommand, log id: 3049d267
>2018-05-30 04:00:03,176-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6444e040] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6444e040, Job ID: 6271370c-476e-40cd-b6e7-39549525dfc1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:00:03,181-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6444e040] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 33a2973b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:00:03,183-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6444e040] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 33a2973b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:00:03,193-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6fdc1260] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:00:03,194-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6fdc1260] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:05:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [2da9d06b] Autorecovering 2 hosts
>2018-05-30 04:05:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [2da9d06b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:05:00,032-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52ba0bfd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:05:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52ba0bfd] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:05:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52ba0bfd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:05:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52ba0bfd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:05:00,040-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [52ba0bfd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50732fed
>2018-05-30 04:05:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [52ba0bfd] FINISH, SetVdsStatusVDSCommand, log id: 50732fed
>2018-05-30 04:05:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52ba0bfd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:05:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52ba0bfd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:05:00,046-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [52ba0bfd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:05:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1fb03873] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:05:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1fb03873] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:05:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1fb03873] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:05:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1fb03873] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:05:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1fb03873] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1888854
>2018-05-30 04:05:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1fb03873] FINISH, SetVdsStatusVDSCommand, log id: 1888854
>2018-05-30 04:05:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [1fb03873] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1afa3106
>2018-05-30 04:05:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [1fb03873] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1afa3106
>2018-05-30 04:05:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1fb03873] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:05:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1fb03873] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:05:00,791-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3c507be8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 612b4bc0
>2018-05-30 04:05:00,792-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [49c1d2ad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 38c722da
>2018-05-30 04:05:00,793-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [49c1d2ad] FINISH, GetHardwareInfoVDSCommand, log id: 38c722da
>2018-05-30 04:05:00,793-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3c507be8] FINISH, GetHardwareInfoVDSCommand, log id: 612b4bc0
>2018-05-30 04:05:00,808-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [621a1033] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:05:00,808-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4340a51e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:05:00,813-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [4f81415d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:05:00,813-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [e7429ee] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:05:00,819-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [393e9446] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:05:00,820-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6812468f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:05:00,832-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6812468f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3308e554
>2018-05-30 04:05:00,832-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [393e9446] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 30219bde
>2018-05-30 04:05:01,209-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6812468f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:05:01,209-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6812468f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3308e554
>2018-05-30 04:05:01,213-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6812468f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:05:01,214-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6812468f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:05:01,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6812468f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:05:01,219-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6812468f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:05:01,219-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [393e9446] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:05:01,219-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [393e9446] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 30219bde
>2018-05-30 04:05:01,222-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6812468f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:05:01,223-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6812468f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:05:01,223-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [393e9446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:05:01,224-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [393e9446] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:05:01,226-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6812468f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:05:01,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [393e9446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:05:01,229-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [393e9446] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:05:01,229-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6812468f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6812468f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:05:01,232-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [393e9446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:05:01,233-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [393e9446] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:05:01,236-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 382608be
>2018-05-30 04:05:01,236-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:05:01,237-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [393e9446] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:05:01,239-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [393e9446] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 393e9446, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:05:01,335-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:05:01,336-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:05:01,336-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:05:01,336-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:05:01,336-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 382608be
>2018-05-30 04:05:01,336-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:05:01,341-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 74b2af8b
>2018-05-30 04:05:01,341-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:05:01,350-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [13c685d3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:05:01,351-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [13c685d3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6f8753db
>2018-05-30 04:05:01,353-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [13c685d3] FINISH, SetVdsStatusVDSCommand, log id: 6f8753db
>2018-05-30 04:05:01,361-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [13c685d3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 13c685d3, Job ID: db55e4df-f2fd-4c05-908e-d98883c7aeb1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:05:01,366-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [13c685d3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4f81415d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:05:01,368-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [13c685d3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4f81415d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:05:01,379-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [f8df71a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:05:01,380-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [f8df71a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:05:01,441-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:05:01,441-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:05:01,441-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:05:01,441-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:05:01,441-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 74b2af8b
>2018-05-30 04:05:01,441-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:05:01,453-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [462589dc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:05:01,455-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [462589dc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41817421
>2018-05-30 04:05:01,457-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [462589dc] FINISH, SetVdsStatusVDSCommand, log id: 41817421
>2018-05-30 04:05:01,462-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [462589dc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 462589dc, Job ID: 6e31eba6-6421-4385-871b-8c8086770d18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:05:01,466-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [462589dc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: e7429ee, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:05:01,468-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [462589dc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: e7429ee, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:05:01,479-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [23f193c5] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:05:01,480-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [23f193c5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4fbc0ad0] Autorecovering 2 hosts
>2018-05-30 04:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4fbc0ad0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:10:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55ad7fe7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55ad7fe7] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55ad7fe7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:10:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55ad7fe7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:10:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [55ad7fe7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b0417ec
>2018-05-30 04:10:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [55ad7fe7] FINISH, SetVdsStatusVDSCommand, log id: 1b0417ec
>2018-05-30 04:10:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55ad7fe7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:10:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55ad7fe7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:10:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [55ad7fe7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:10:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4f70d84f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:10:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4f70d84f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:10:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4f70d84f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:10:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4f70d84f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:10:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4f70d84f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 656da39e
>2018-05-30 04:10:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4f70d84f] FINISH, SetVdsStatusVDSCommand, log id: 656da39e
>2018-05-30 04:10:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [4f70d84f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 99567f0
>2018-05-30 04:10:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [4f70d84f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 99567f0
>2018-05-30 04:10:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4f70d84f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:10:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4f70d84f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:10:02,139-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [50bc1f2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 43d71b54
>2018-05-30 04:10:02,141-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [50bc1f2] FINISH, GetHardwareInfoVDSCommand, log id: 43d71b54
>2018-05-30 04:10:02,154-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [426cf436] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:10:02,163-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [448770ad] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:10:02,166-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5ad1c116] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3dd00207
>2018-05-30 04:10:02,168-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5ad1c116] FINISH, GetHardwareInfoVDSCommand, log id: 3dd00207
>2018-05-30 04:10:02,169-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7c579a2e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:10:02,182-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7c579a2e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1102f6b7
>2018-05-30 04:10:02,202-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6846b247] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:10:02,210-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [2903212e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:10:02,215-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [137a4b8b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:10:02,270-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [137a4b8b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 11ee6065
>2018-05-30 04:10:02,564-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7c579a2e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:10:02,564-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7c579a2e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1102f6b7
>2018-05-30 04:10:02,568-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7c579a2e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:10:02,568-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7c579a2e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:10:02,572-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7c579a2e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:10:02,573-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7c579a2e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:10:02,577-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7c579a2e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:10:02,577-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7c579a2e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:10:02,580-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7c579a2e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:10:02,582-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7c579a2e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7c579a2e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:10:02,588-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 315153fc
>2018-05-30 04:10:02,588-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:10:02,634-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [137a4b8b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:10:02,634-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [137a4b8b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 11ee6065
>2018-05-30 04:10:02,638-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [137a4b8b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:10:02,638-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [137a4b8b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:10:02,642-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [137a4b8b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:10:02,643-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [137a4b8b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:10:02,646-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [137a4b8b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:10:02,647-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [137a4b8b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:10:02,650-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [137a4b8b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:10:02,652-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [137a4b8b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 137a4b8b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:10:02,689-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-11) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:10:02,690-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:10:02,690-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:10:02,690-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:10:02,690-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] FINISH, ConnectStoragePoolVDSCommand, log id: 315153fc
>2018-05-30 04:10:02,690-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-11) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:10:02,694-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 37f024a0
>2018-05-30 04:10:02,694-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:10:02,703-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [52c49a8c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:10:02,704-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [52c49a8c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d448496
>2018-05-30 04:10:02,706-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [52c49a8c] FINISH, SetVdsStatusVDSCommand, log id: d448496
>2018-05-30 04:10:02,740-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [52c49a8c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 52c49a8c, Job ID: 34282f02-5e86-4d92-b0ff-6af24c10632b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:10:02,744-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [52c49a8c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 448770ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:10:02,746-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [52c49a8c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 448770ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:10:02,756-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [49e8078d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:10:02,757-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [49e8078d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:10:02,791-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:10:02,791-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:10:02,791-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:10:02,791-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:10:02,791-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 37f024a0
>2018-05-30 04:10:02,791-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:10:02,803-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [6fad4745] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:10:02,805-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6fad4745] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4d6ea658
>2018-05-30 04:10:02,807-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6fad4745] FINISH, SetVdsStatusVDSCommand, log id: 4d6ea658
>2018-05-30 04:10:02,812-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6fad4745] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6fad4745, Job ID: 3e73e2a0-308a-4f63-88e9-3a5280c54174, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:10:02,816-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6fad4745] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2903212e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:10:02,818-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6fad4745] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2903212e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:10:02,829-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [65f72f04] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:10:02,830-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [65f72f04] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:15:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [708bf037] Autorecovering 2 hosts
>2018-05-30 04:15:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [708bf037] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:15:00,015-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5a0a7e9b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:15:00,021-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5a0a7e9b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:15:00,021-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5a0a7e9b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:15:00,021-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5a0a7e9b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:15:00,023-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5a0a7e9b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 11ca1de7
>2018-05-30 04:15:00,026-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5a0a7e9b] FINISH, SetVdsStatusVDSCommand, log id: 11ca1de7
>2018-05-30 04:15:00,028-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5a0a7e9b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:15:00,029-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5a0a7e9b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:15:00,029-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5a0a7e9b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:15:00,035-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65214051] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:15:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65214051] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:15:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65214051] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:15:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65214051] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:15:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [65214051] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: e83708a
>2018-05-30 04:15:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [65214051] FINISH, SetVdsStatusVDSCommand, log id: e83708a
>2018-05-30 04:15:00,047-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [65214051] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 12495ee3
>2018-05-30 04:15:00,049-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [65214051] FINISH, SetHaMaintenanceModeVDSCommand, log id: 12495ee3
>2018-05-30 04:15:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65214051] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:15:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65214051] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:15:00,396-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [327d07af] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5bba3aae
>2018-05-30 04:15:00,398-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [327d07af] FINISH, GetHardwareInfoVDSCommand, log id: 5bba3aae
>2018-05-30 04:15:00,411-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [18026d64] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:15:00,419-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [b492edc] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:15:00,424-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3ded4261] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:15:00,435-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3ded4261] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3bc609e7
>2018-05-30 04:15:00,483-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7b2b4093] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 60c2ddc7
>2018-05-30 04:15:00,485-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7b2b4093] FINISH, GetHardwareInfoVDSCommand, log id: 60c2ddc7
>2018-05-30 04:15:00,498-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [3b050204] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:15:00,502-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [2bbfe039] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:15:00,508-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [783c8ff5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:15:00,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [783c8ff5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 64547ed8
>2018-05-30 04:15:00,816-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3ded4261] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:15:00,816-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3ded4261] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3bc609e7
>2018-05-30 04:15:00,820-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3ded4261] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:15:00,821-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3ded4261] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:15:00,824-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3ded4261] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:15:00,825-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3ded4261] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:15:00,829-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3ded4261] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:15:00,830-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3ded4261] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:15:00,833-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3ded4261] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:15:00,835-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3ded4261] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3ded4261, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:15:00,842-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5ba74413
>2018-05-30 04:15:00,842-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:15:00,885-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [783c8ff5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:15:00,885-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [783c8ff5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 64547ed8
>2018-05-30 04:15:00,888-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [783c8ff5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:15:00,889-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [783c8ff5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:15:00,893-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [783c8ff5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:15:00,894-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [783c8ff5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:15:00,898-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [783c8ff5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:15:00,898-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [783c8ff5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:15:00,901-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [783c8ff5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:15:00,904-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [783c8ff5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 783c8ff5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:15:00,942-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:15:00,942-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:15:00,942-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:15:00,942-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:15:00,942-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5ba74413
>2018-05-30 04:15:00,942-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:15:00,943-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5b2ce322
>2018-05-30 04:15:00,943-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:15:00,955-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [32086e53] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:15:00,956-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [32086e53] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1639db2
>2018-05-30 04:15:00,959-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [32086e53] FINISH, SetVdsStatusVDSCommand, log id: 1639db2
>2018-05-30 04:15:00,964-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [32086e53] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 32086e53, Job ID: 2ebbdecf-e8de-4cf9-9e66-6c060c73e165, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:15:00,969-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [32086e53] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: b492edc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:15:00,971-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [32086e53] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: b492edc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:15:00,981-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [472dc71] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:15:00,982-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [472dc71] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:15:01,044-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:15:01,044-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:15:01,044-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:15:01,044-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:15:01,044-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5b2ce322
>2018-05-30 04:15:01,044-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:15:01,058-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [29c311ed] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:15:01,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [29c311ed] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 20609f6d
>2018-05-30 04:15:01,061-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [29c311ed] FINISH, SetVdsStatusVDSCommand, log id: 20609f6d
>2018-05-30 04:15:01,066-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [29c311ed] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 29c311ed, Job ID: c36cc375-3508-4578-9d66-7a0a52377af0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:15:01,071-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [29c311ed] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2bbfe039, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:15:01,073-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [29c311ed] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2bbfe039, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:15:01,083-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [70c8ffc6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:15:01,084-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [70c8ffc6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:20:00,014-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [37261055] Autorecovering 2 hosts
>2018-05-30 04:20:00,014-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [37261055] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [702d3e47] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:20:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [702d3e47] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:20:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [702d3e47] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:20:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [702d3e47] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:20:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [702d3e47] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 756192df
>2018-05-30 04:20:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [702d3e47] FINISH, SetVdsStatusVDSCommand, log id: 756192df
>2018-05-30 04:20:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [702d3e47] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:20:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [702d3e47] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:20:00,082-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [702d3e47] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:20:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5cc5b73d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:20:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5cc5b73d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:20:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5cc5b73d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:20:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5cc5b73d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:20:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5cc5b73d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72387a8c
>2018-05-30 04:20:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5cc5b73d] FINISH, SetVdsStatusVDSCommand, log id: 72387a8c
>2018-05-30 04:20:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5cc5b73d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1a70c229
>2018-05-30 04:20:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5cc5b73d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1a70c229
>2018-05-30 04:20:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5cc5b73d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:20:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5cc5b73d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:20:01,633-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [54950fc6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6c79e364
>2018-05-30 04:20:01,635-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [54950fc6] FINISH, GetHardwareInfoVDSCommand, log id: 6c79e364
>2018-05-30 04:20:01,647-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [56f5f53f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:20:01,656-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5076460d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:20:01,661-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6d238e42] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:20:01,672-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6d238e42] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5450cdf0
>2018-05-30 04:20:01,687-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [27d2151d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7fbe57ba
>2018-05-30 04:20:01,689-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [27d2151d] FINISH, GetHardwareInfoVDSCommand, log id: 7fbe57ba
>2018-05-30 04:20:01,701-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [6c278d87] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:20:01,706-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [475b8188] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:20:01,711-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1f22534a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:20:01,717-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1f22534a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 26a6a718
>2018-05-30 04:20:02,059-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d238e42] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:20:02,059-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6d238e42] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5450cdf0
>2018-05-30 04:20:02,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d238e42] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:20:02,064-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6d238e42] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:20:02,068-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d238e42] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:20:02,068-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6d238e42] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:20:02,072-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d238e42] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:20:02,073-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6d238e42] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:20:02,076-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6d238e42] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:20:02,079-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d238e42] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6d238e42, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:20:02,085-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 31cdf395
>2018-05-30 04:20:02,085-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:20:02,182-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:20:02,182-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:20:02,182-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:20:02,182-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:20:02,182-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 31cdf395
>2018-05-30 04:20:02,182-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:20:02,195-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [b513064] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:20:02,196-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [b513064] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 62b65f75
>2018-05-30 04:20:02,198-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [b513064] FINISH, SetVdsStatusVDSCommand, log id: 62b65f75
>2018-05-30 04:20:02,231-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [b513064] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: b513064, Job ID: df47f7a6-0b11-44ad-a3d0-667499a8a838, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:20:02,236-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [b513064] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5076460d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:20:02,238-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [b513064] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5076460d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:20:02,249-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [bd8d490] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:20:02,250-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [bd8d490] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:20:03,087-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1f22534a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:20:03,087-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1f22534a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 26a6a718
>2018-05-30 04:20:03,091-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1f22534a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:20:03,092-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1f22534a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:20:03,096-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1f22534a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:20:03,097-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1f22534a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:20:03,100-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1f22534a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:20:03,101-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1f22534a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:20:03,104-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1f22534a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:20:03,107-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1f22534a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1f22534a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:20:03,110-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 57a73ca4
>2018-05-30 04:20:03,110-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:20:03,206-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:20:03,206-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:20:03,206-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:20:03,206-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:20:03,206-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 57a73ca4
>2018-05-30 04:20:03,206-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:20:03,219-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [4f4d468] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:20:03,221-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4f4d468] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 108b33ff
>2018-05-30 04:20:03,223-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4f4d468] FINISH, SetVdsStatusVDSCommand, log id: 108b33ff
>2018-05-30 04:20:03,228-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4f4d468] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4f4d468, Job ID: d923d9b1-fcf1-4619-9f5b-170c270c69d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:20:03,233-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4f4d468] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 475b8188, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:20:03,235-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4f4d468] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 475b8188, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:20:03,246-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [44ffae5e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:20:03,247-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [44ffae5e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [3bac3663] Autorecovering 2 hosts
>2018-05-30 04:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [3bac3663] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:25:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6ca414ed] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6ca414ed] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6ca414ed] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:25:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6ca414ed] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:25:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6ca414ed] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 79005363
>2018-05-30 04:25:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6ca414ed] FINISH, SetVdsStatusVDSCommand, log id: 79005363
>2018-05-30 04:25:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6ca414ed] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:25:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6ca414ed] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:25:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [6ca414ed] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:25:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [555a2afb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:25:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [555a2afb] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:25:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [555a2afb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:25:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [555a2afb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:25:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [555a2afb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7ca3a8b6
>2018-05-30 04:25:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [555a2afb] FINISH, SetVdsStatusVDSCommand, log id: 7ca3a8b6
>2018-05-30 04:25:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [555a2afb] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 790ec2ce
>2018-05-30 04:25:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [555a2afb] FINISH, SetHaMaintenanceModeVDSCommand, log id: 790ec2ce
>2018-05-30 04:25:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [555a2afb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:25:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [555a2afb] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:25:00,939-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3bcab9f6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 77d6f6
>2018-05-30 04:25:00,941-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3bcab9f6] FINISH, GetHardwareInfoVDSCommand, log id: 77d6f6
>2018-05-30 04:25:00,954-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2a8881f1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:25:00,963-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [52ca6e4a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:25:00,968-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [48bded5f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:25:00,979-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [48bded5f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 24d4171f
>2018-05-30 04:25:01,349-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [48bded5f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:25:01,349-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [48bded5f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 24d4171f
>2018-05-30 04:25:01,353-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [48bded5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:25:01,354-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [48bded5f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:25:01,357-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [48bded5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:25:01,358-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [48bded5f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:25:01,362-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [48bded5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:25:01,362-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [48bded5f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:25:01,365-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [48bded5f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:25:01,367-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [48bded5f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 48bded5f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:25:01,374-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5eda525b
>2018-05-30 04:25:01,374-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:25:01,475-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:25:01,475-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:25:01,475-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:25:01,475-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:25:01,475-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5eda525b
>2018-05-30 04:25:01,475-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:25:01,489-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [12599ac1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:25:01,490-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [12599ac1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1edb3265
>2018-05-30 04:25:01,493-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [12599ac1] FINISH, SetVdsStatusVDSCommand, log id: 1edb3265
>2018-05-30 04:25:01,527-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12599ac1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 12599ac1, Job ID: 8ad014b6-f993-4776-aeaa-3df72582529e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:25:01,532-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12599ac1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 52ca6e4a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:25:01,534-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12599ac1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 52ca6e4a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:25:01,545-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [ed36b0a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:25:01,546-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [ed36b0a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:25:02,998-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [44883bce] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 798466f6
>2018-05-30 04:25:03,000-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [44883bce] FINISH, GetHardwareInfoVDSCommand, log id: 798466f6
>2018-05-30 04:25:03,012-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [f32c9d7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:25:03,016-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [5ffbc3c4] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:25:03,021-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2f68b714] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:25:03,028-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2f68b714] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 246c0c20
>2018-05-30 04:25:03,399-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2f68b714] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:25:03,399-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2f68b714] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 246c0c20
>2018-05-30 04:25:03,402-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2f68b714] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:25:03,403-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2f68b714] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:25:03,407-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2f68b714] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:25:03,408-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2f68b714] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:25:03,411-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2f68b714] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:25:03,412-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2f68b714] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:25:03,415-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2f68b714] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:25:03,417-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2f68b714] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2f68b714, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:25:03,420-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 59ee69c8
>2018-05-30 04:25:03,420-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:25:03,515-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:25:03,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:25:03,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:25:03,515-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:25:03,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 59ee69c8
>2018-05-30 04:25:03,515-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:25:03,527-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [29a4481b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:25:03,529-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [29a4481b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 781f4662
>2018-05-30 04:25:03,531-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [29a4481b] FINISH, SetVdsStatusVDSCommand, log id: 781f4662
>2018-05-30 04:25:03,536-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [29a4481b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 29a4481b, Job ID: 1dd2c3e3-6da4-4991-8af6-33d62a6dce55, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:25:03,540-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [29a4481b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5ffbc3c4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:25:03,542-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [29a4481b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5ffbc3c4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:25:03,554-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [83e707e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:25:03,555-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [83e707e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [8dbca0f] Autorecovering 2 hosts
>2018-05-30 04:30:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [8dbca0f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:30:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [ceede11] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:30:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [ceede11] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:30:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [ceede11] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:30:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [ceede11] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:30:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [ceede11] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 74e25ede
>2018-05-30 04:30:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [ceede11] FINISH, SetVdsStatusVDSCommand, log id: 74e25ede
>2018-05-30 04:30:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [ceede11] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:30:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [ceede11] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:30:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [ceede11] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:30:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [781ad20a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:30:00,111-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [781ad20a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:30:00,111-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [781ad20a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:30:00,111-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [781ad20a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:30:00,112-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [781ad20a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a6e09b9
>2018-05-30 04:30:00,114-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [781ad20a] FINISH, SetVdsStatusVDSCommand, log id: a6e09b9
>2018-05-30 04:30:00,117-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [781ad20a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 13929c83
>2018-05-30 04:30:00,119-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [781ad20a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 13929c83
>2018-05-30 04:30:00,119-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [781ad20a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:30:00,121-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [781ad20a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:30:01,156-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [33adfc71] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 44bba2b4
>2018-05-30 04:30:01,157-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [33adfc71] FINISH, GetHardwareInfoVDSCommand, log id: 44bba2b4
>2018-05-30 04:30:01,170-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4fb181a6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:30:01,174-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [2aab5a74] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:30:01,180-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [62354ff5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:30:01,192-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [62354ff5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 79d0636f
>2018-05-30 04:30:01,570-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62354ff5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:30:01,570-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [62354ff5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 79d0636f
>2018-05-30 04:30:01,574-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62354ff5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:30:01,575-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [62354ff5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:30:01,579-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62354ff5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:30:01,579-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [62354ff5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:30:01,583-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62354ff5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:30:01,584-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [62354ff5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:30:01,587-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [62354ff5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:30:01,590-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62354ff5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 62354ff5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:30:01,597-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 70d6ce9c
>2018-05-30 04:30:01,597-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:30:01,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:30:01,695-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:30:01,695-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:30:01,695-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:30:01,695-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 70d6ce9c
>2018-05-30 04:30:01,695-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:30:01,709-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [26d2818d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:30:01,710-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [26d2818d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: bd021fe
>2018-05-30 04:30:01,712-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [26d2818d] FINISH, SetVdsStatusVDSCommand, log id: bd021fe
>2018-05-30 04:30:01,720-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26d2818d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 26d2818d, Job ID: b9fa3cdd-a100-46cd-8d3f-8d7b59427c91, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:30:01,725-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26d2818d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2aab5a74, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:30:01,727-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26d2818d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2aab5a74, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:30:01,737-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [4e8e771] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:30:01,738-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [4e8e771] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:30:02,321-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [590925d7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: ae79720
>2018-05-30 04:30:02,323-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [590925d7] FINISH, GetHardwareInfoVDSCommand, log id: ae79720
>2018-05-30 04:30:02,336-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [3bf189ce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:30:02,345-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [66c0d002] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:30:02,350-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5fd8bfbf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:30:02,361-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5fd8bfbf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3e5bdebd
>2018-05-30 04:30:04,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5fd8bfbf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:30:04,680-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5fd8bfbf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3e5bdebd
>2018-05-30 04:30:04,684-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5fd8bfbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:30:04,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5fd8bfbf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:30:04,688-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5fd8bfbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:30:04,689-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5fd8bfbf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:30:04,693-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5fd8bfbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:30:04,694-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5fd8bfbf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:30:04,696-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5fd8bfbf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:30:04,699-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5fd8bfbf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5fd8bfbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:30:04,705-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 54e2e234
>2018-05-30 04:30:04,705-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:30:04,801-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:30:04,801-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:30:04,801-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:30:04,801-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:30:04,801-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 54e2e234
>2018-05-30 04:30:04,801-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:30:04,815-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [36e66a1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:30:04,816-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [36e66a1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5ab36235
>2018-05-30 04:30:04,818-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [36e66a1] FINISH, SetVdsStatusVDSCommand, log id: 5ab36235
>2018-05-30 04:30:04,850-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [36e66a1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 36e66a1, Job ID: be4fe462-312d-45fe-873f-dff6397383fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:30:04,855-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [36e66a1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 66c0d002, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:30:04,857-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [36e66a1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 66c0d002, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:30:04,868-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [7da1b540] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:30:04,869-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [7da1b540] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [184f095] Autorecovering 2 hosts
>2018-05-30 04:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [184f095] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:35:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4903a70b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4903a70b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4903a70b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4903a70b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:35:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4903a70b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a9f070
>2018-05-30 04:35:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4903a70b] FINISH, SetVdsStatusVDSCommand, log id: a9f070
>2018-05-30 04:35:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4903a70b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:35:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4903a70b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:35:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [4903a70b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:35:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [59986535] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:35:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [59986535] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:35:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [59986535] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:35:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [59986535] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:35:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [59986535] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6968bbec
>2018-05-30 04:35:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [59986535] FINISH, SetVdsStatusVDSCommand, log id: 6968bbec
>2018-05-30 04:35:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [59986535] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2e637d0c
>2018-05-30 04:35:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [59986535] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2e637d0c
>2018-05-30 04:35:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [59986535] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:35:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [59986535] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:35:02,390-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3f008328] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 43e806ba
>2018-05-30 04:35:02,392-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3f008328] FINISH, GetHardwareInfoVDSCommand, log id: 43e806ba
>2018-05-30 04:35:02,404-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [615cb22e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:35:02,413-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1fcac043] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:35:02,418-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [4c47655a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:35:02,430-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [4c47655a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 25332071
>2018-05-30 04:35:02,519-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [169b6782] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 367c4955
>2018-05-30 04:35:02,520-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [169b6782] FINISH, GetHardwareInfoVDSCommand, log id: 367c4955
>2018-05-30 04:35:02,532-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [3150491b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:35:02,537-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [110b08af] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:35:02,541-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [169c2a24] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:35:02,547-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [169c2a24] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 56226c08
>2018-05-30 04:35:02,809-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c47655a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:35:02,809-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [4c47655a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 25332071
>2018-05-30 04:35:02,812-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c47655a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:35:02,813-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [4c47655a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:35:02,817-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c47655a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:35:02,818-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [4c47655a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:35:02,821-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c47655a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:35:02,822-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [4c47655a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:35:02,825-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [4c47655a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:35:02,827-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c47655a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4c47655a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:35:02,834-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 270df9bf
>2018-05-30 04:35:02,834-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:35:02,913-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169c2a24] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:35:02,913-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [169c2a24] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 56226c08
>2018-05-30 04:35:02,917-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169c2a24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:35:02,918-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [169c2a24] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:35:02,922-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169c2a24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:35:02,922-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [169c2a24] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:35:02,926-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169c2a24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:35:02,926-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [169c2a24] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:35:02,929-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [169c2a24] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:35:02,931-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169c2a24] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 169c2a24, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:35:02,934-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:35:02,934-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:35:02,934-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:35:02,934-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:35:02,934-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 270df9bf
>2018-05-30 04:35:02,934-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:35:02,936-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7b030ba3
>2018-05-30 04:35:02,936-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:35:02,947-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [419e5b08] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:35:02,948-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [419e5b08] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 768740c2
>2018-05-30 04:35:02,950-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [419e5b08] FINISH, SetVdsStatusVDSCommand, log id: 768740c2
>2018-05-30 04:35:02,983-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e5b08] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 419e5b08, Job ID: 520e086a-a409-4129-9153-17799cecb3a6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:35:02,987-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e5b08] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1fcac043, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:35:02,990-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e5b08] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1fcac043, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:35:03,000-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [10f9c677] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:35:03,001-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [10f9c677] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:35:03,035-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:35:03,035-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:35:03,035-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:35:03,035-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:35:03,035-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7b030ba3
>2018-05-30 04:35:03,035-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:35:03,047-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [803795f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:35:03,048-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [803795f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 549f168f
>2018-05-30 04:35:03,050-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [803795f] FINISH, SetVdsStatusVDSCommand, log id: 549f168f
>2018-05-30 04:35:03,055-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [803795f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 803795f, Job ID: 9df76752-927d-4e1f-9437-5021074f23e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:35:03,060-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [803795f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 110b08af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:35:03,061-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [803795f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 110b08af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:35:03,071-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [43a9d28e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:35:03,073-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [43a9d28e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:40:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6a7e41f5] Autorecovering 2 hosts
>2018-05-30 04:40:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6a7e41f5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:40:00,030-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34f3eaf5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:40:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34f3eaf5] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:40:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34f3eaf5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:40:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34f3eaf5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:40:00,039-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [34f3eaf5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45ddbab1
>2018-05-30 04:40:00,041-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [34f3eaf5] FINISH, SetVdsStatusVDSCommand, log id: 45ddbab1
>2018-05-30 04:40:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34f3eaf5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:40:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34f3eaf5] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:40:00,044-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [34f3eaf5] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:40:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6da4218b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:40:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6da4218b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:40:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6da4218b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:40:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6da4218b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:40:00,057-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6da4218b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32b35b71
>2018-05-30 04:40:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6da4218b] FINISH, SetVdsStatusVDSCommand, log id: 32b35b71
>2018-05-30 04:40:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6da4218b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4936e6c1
>2018-05-30 04:40:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6da4218b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4936e6c1
>2018-05-30 04:40:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6da4218b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:40:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6da4218b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:40:00,585-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4c885a9b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4a448545
>2018-05-30 04:40:00,587-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4c885a9b] FINISH, GetHardwareInfoVDSCommand, log id: 4a448545
>2018-05-30 04:40:00,600-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [1fe9f898] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:40:00,609-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [741b389c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:40:00,615-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [3365f2cb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:40:00,627-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [3365f2cb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 35192b8b
>2018-05-30 04:40:00,762-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [55b13917] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 24b43c45
>2018-05-30 04:40:00,764-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [55b13917] FINISH, GetHardwareInfoVDSCommand, log id: 24b43c45
>2018-05-30 04:40:00,776-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [7ab8690] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:40:00,780-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6e210b9a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:40:00,785-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5b3e7680] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:40:00,791-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5b3e7680] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 45ca033c
>2018-05-30 04:40:01,005-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3365f2cb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:40:01,005-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [3365f2cb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 35192b8b
>2018-05-30 04:40:01,009-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3365f2cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:40:01,010-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [3365f2cb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:40:01,014-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3365f2cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:40:01,015-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [3365f2cb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:40:01,018-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3365f2cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:40:01,019-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [3365f2cb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:40:01,022-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [3365f2cb] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:40:01,025-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3365f2cb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3365f2cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:40:01,031-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5a55e6f1
>2018-05-30 04:40:01,031-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:40:01,125-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:40:01,125-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:40:01,125-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:40:01,125-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:40:01,125-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5a55e6f1
>2018-05-30 04:40:01,125-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:40:01,138-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [714c20d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:40:01,139-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [714c20d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 634de580
>2018-05-30 04:40:01,142-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [714c20d] FINISH, SetVdsStatusVDSCommand, log id: 634de580
>2018-05-30 04:40:01,175-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [714c20d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 714c20d, Job ID: 5a5e4da1-aecd-4ab3-bd40-58d71d95a653, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:40:01,180-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [714c20d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 741b389c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:40:01,182-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [714c20d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 741b389c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:40:01,184-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b3e7680] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:40:01,184-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5b3e7680] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 45ca033c
>2018-05-30 04:40:01,188-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b3e7680] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:40:01,189-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5b3e7680] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:40:01,193-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b3e7680] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:40:01,193-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5b3e7680] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:40:01,195-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [1eaf9c5f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:40:01,197-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [1eaf9c5f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:40:01,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b3e7680] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:40:01,199-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5b3e7680] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:40:01,251-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5b3e7680] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:40:01,255-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b3e7680] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5b3e7680, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:40:01,261-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 35ff52e
>2018-05-30 04:40:01,261-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:40:01,362-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:40:01,362-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:40:01,362-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:40:01,362-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:40:01,362-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 35ff52e
>2018-05-30 04:40:01,362-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:40:01,375-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [3c95ccd2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:40:01,376-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3c95ccd2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54d7c7d8
>2018-05-30 04:40:01,378-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3c95ccd2] FINISH, SetVdsStatusVDSCommand, log id: 54d7c7d8
>2018-05-30 04:40:01,410-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c95ccd2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c95ccd2, Job ID: 9e659b23-a2a6-4a4b-8015-624a26701f9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:40:01,415-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c95ccd2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6e210b9a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:40:01,417-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c95ccd2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6e210b9a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:40:01,428-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [178d8b4b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:40:01,429-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [178d8b4b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1bb7d759] Autorecovering 2 hosts
>2018-05-30 04:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1bb7d759] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:45:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [75f48b1c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [75f48b1c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [75f48b1c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [75f48b1c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:45:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [75f48b1c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d47b525
>2018-05-30 04:45:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [75f48b1c] FINISH, SetVdsStatusVDSCommand, log id: d47b525
>2018-05-30 04:45:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [75f48b1c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:45:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [75f48b1c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:45:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [75f48b1c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:45:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [15094055] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [15094055] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [15094055] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [15094055] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:45:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [15094055] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2fee576a
>2018-05-30 04:45:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [15094055] FINISH, SetVdsStatusVDSCommand, log id: 2fee576a
>2018-05-30 04:45:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [15094055] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5c09eeb5
>2018-05-30 04:45:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [15094055] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5c09eeb5
>2018-05-30 04:45:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [15094055] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:45:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [15094055] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:45:01,833-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3837e18a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 416c44b8
>2018-05-30 04:45:01,835-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3837e18a] FINISH, GetHardwareInfoVDSCommand, log id: 416c44b8
>2018-05-30 04:45:01,849-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [70a7ed83] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:45:01,858-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [50873fb9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:45:01,863-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2107b584] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:45:01,875-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2107b584] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23818806
>2018-05-30 04:45:02,088-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4371efc0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1ef2cd12
>2018-05-30 04:45:02,090-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4371efc0] FINISH, GetHardwareInfoVDSCommand, log id: 1ef2cd12
>2018-05-30 04:45:02,104-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [60079d89] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:45:02,109-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [138ea316] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:45:02,114-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1a25153] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:45:02,122-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1a25153] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6001bb62
>2018-05-30 04:45:02,254-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2107b584] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:45:02,254-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2107b584] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 23818806
>2018-05-30 04:45:02,258-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2107b584] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:45:02,259-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2107b584] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:45:02,263-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2107b584] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:45:02,263-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2107b584] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:45:02,267-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2107b584] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:45:02,268-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2107b584] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:45:02,271-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2107b584] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:45:02,273-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2107b584] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2107b584, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:45:02,281-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 793a0f45
>2018-05-30 04:45:02,281-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:45:02,384-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:45:02,384-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:45:02,384-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:45:02,384-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:45:02,384-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 793a0f45
>2018-05-30 04:45:02,384-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:45:02,398-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [51c09498] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:45:02,399-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [51c09498] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8017fa4
>2018-05-30 04:45:02,401-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [51c09498] FINISH, SetVdsStatusVDSCommand, log id: 8017fa4
>2018-05-30 04:45:02,410-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [51c09498] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 51c09498, Job ID: 06d2b49d-79b1-46b8-beb1-452aa16ebb6a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:45:02,415-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [51c09498] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 50873fb9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:45:02,417-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [51c09498] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 50873fb9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:45:02,428-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [7aa0d99e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:45:02,429-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7aa0d99e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:45:02,492-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a25153] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:45:02,492-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1a25153] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6001bb62
>2018-05-30 04:45:02,496-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a25153] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:45:02,497-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1a25153] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:45:02,501-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a25153] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:45:02,501-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1a25153] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:45:02,505-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a25153] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:45:02,506-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1a25153] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:45:02,509-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1a25153] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:45:02,512-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a25153] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1a25153, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:45:02,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4e2e29d8
>2018-05-30 04:45:02,515-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:45:02,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:45:02,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:45:02,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:45:02,614-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:45:02,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4e2e29d8
>2018-05-30 04:45:02,614-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:45:02,628-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [490fb97c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:45:02,629-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [490fb97c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ca3624e
>2018-05-30 04:45:02,631-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [490fb97c] FINISH, SetVdsStatusVDSCommand, log id: 6ca3624e
>2018-05-30 04:45:02,637-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490fb97c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 490fb97c, Job ID: 4edb7719-4583-485d-9246-4e651a68f416, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:45:02,641-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490fb97c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 138ea316, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:45:02,643-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490fb97c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 138ea316, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:45:02,653-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [228f4d26] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:45:02,654-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [228f4d26] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:50:00,015-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4cd6adb6] Autorecovering 2 hosts
>2018-05-30 04:50:00,015-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4cd6adb6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:50:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17dc4c37] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:50:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17dc4c37] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:50:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17dc4c37] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:50:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17dc4c37] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:50:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [17dc4c37] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6164ae7c
>2018-05-30 04:50:00,080-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [17dc4c37] FINISH, SetVdsStatusVDSCommand, log id: 6164ae7c
>2018-05-30 04:50:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17dc4c37] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:50:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17dc4c37] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:50:00,083-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [17dc4c37] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:50:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5721c550] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:50:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5721c550] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:50:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5721c550] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:50:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5721c550] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:50:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5721c550] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7796ba86
>2018-05-30 04:50:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5721c550] FINISH, SetVdsStatusVDSCommand, log id: 7796ba86
>2018-05-30 04:50:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [5721c550] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 38be75a2
>2018-05-30 04:50:00,103-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [5721c550] FINISH, SetHaMaintenanceModeVDSCommand, log id: 38be75a2
>2018-05-30 04:50:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5721c550] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:50:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5721c550] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:50:00,337-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [33ae8956] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 195f9b5d
>2018-05-30 04:50:00,338-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [33ae8956] FINISH, GetHardwareInfoVDSCommand, log id: 195f9b5d
>2018-05-30 04:50:00,351-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2b447a48] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:50:00,359-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4774af54] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:50:00,365-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [5298c26a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:50:00,371-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [5298c26a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5e23b36f
>2018-05-30 04:50:00,746-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5298c26a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:50:00,746-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [5298c26a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5e23b36f
>2018-05-30 04:50:00,750-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5298c26a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:50:00,751-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5298c26a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:50:00,755-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5298c26a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:50:00,755-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5298c26a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:50:00,759-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5298c26a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:50:00,760-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5298c26a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:50:00,763-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [5298c26a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:50:00,765-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5298c26a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5298c26a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:50:00,768-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a45660c
>2018-05-30 04:50:00,768-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:50:00,868-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:50:00,868-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:50:00,868-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:50:00,868-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:50:00,868-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a45660c
>2018-05-30 04:50:00,868-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:50:00,882-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [1d95b2af] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:50:00,883-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1d95b2af] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b20f12c
>2018-05-30 04:50:00,885-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1d95b2af] FINISH, SetVdsStatusVDSCommand, log id: 1b20f12c
>2018-05-30 04:50:00,915-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d95b2af] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1d95b2af, Job ID: 99c79e08-2182-4965-9040-22bb18303511, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:50:00,920-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d95b2af] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4774af54, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:50:00,922-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d95b2af] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4774af54, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:50:00,933-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [443eaf2e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:50:00,934-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [443eaf2e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:50:03,154-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [78f29ea9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 285697e8
>2018-05-30 04:50:03,155-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [78f29ea9] FINISH, GetHardwareInfoVDSCommand, log id: 285697e8
>2018-05-30 04:50:03,168-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [58d4986d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:50:03,172-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [7acf7630] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:50:03,177-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6036e5be] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:50:03,184-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6036e5be] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7563918d
>2018-05-30 04:50:03,563-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6036e5be] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:50:03,563-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6036e5be] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7563918d
>2018-05-30 04:50:03,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6036e5be] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:50:03,567-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6036e5be] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:50:03,571-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6036e5be] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:50:03,572-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6036e5be] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:50:03,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6036e5be] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:50:03,576-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6036e5be] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:50:03,579-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6036e5be] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:50:03,581-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6036e5be] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6036e5be, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:50:03,584-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2264770b
>2018-05-30 04:50:03,584-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:50:03,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:50:03,682-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:50:03,682-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:50:03,682-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:50:03,682-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2264770b
>2018-05-30 04:50:03,682-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:50:03,695-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [243013a9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:50:03,696-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [243013a9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31d5641e
>2018-05-30 04:50:03,698-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [243013a9] FINISH, SetVdsStatusVDSCommand, log id: 31d5641e
>2018-05-30 04:50:03,703-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [243013a9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 243013a9, Job ID: 54fc9c71-ba3a-4a3f-8b93-80484f0eba74, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:50:03,708-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [243013a9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7acf7630, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:50:03,710-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [243013a9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7acf7630, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:50:03,720-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [4a255bef] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:50:03,721-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [4a255bef] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:54:43,544-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler6) [3c038fa3] Attempting to update VMs/Templates Ovf.
>2018-05-30 04:54:43,547-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 04:54:43,547-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 04:54:43,547-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:54:43,552-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-30 04:54:43,555-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-30 04:54:43,555-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Attempting to update template OVFs in Data Center 'Default'
>2018-05-30 04:54:43,555-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-30 04:54:43,555-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 04:54:43,556-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 04:54:43,556-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 04:55:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [71b42b9c] Autorecovering 2 hosts
>2018-05-30 04:55:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [71b42b9c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:55:00,011-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [69e706c1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:55:00,017-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [69e706c1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:55:00,017-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [69e706c1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:55:00,017-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [69e706c1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:55:00,019-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [69e706c1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 759ce807
>2018-05-30 04:55:00,021-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [69e706c1] FINISH, SetVdsStatusVDSCommand, log id: 759ce807
>2018-05-30 04:55:00,022-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [69e706c1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:55:00,024-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [69e706c1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 04:55:00,024-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [69e706c1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:55:00,029-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [13053773] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:55:00,035-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [13053773] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 04:55:00,035-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [13053773] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:55:00,035-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [13053773] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:55:00,037-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [13053773] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 134b8098
>2018-05-30 04:55:00,039-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [13053773] FINISH, SetVdsStatusVDSCommand, log id: 134b8098
>2018-05-30 04:55:00,041-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [13053773] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 51b4347a
>2018-05-30 04:55:00,043-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [13053773] FINISH, SetHaMaintenanceModeVDSCommand, log id: 51b4347a
>2018-05-30 04:55:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [13053773] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 04:55:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [13053773] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 04:55:01,324-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [53c4f6df] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7308c50c
>2018-05-30 04:55:01,326-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [53c4f6df] FINISH, GetHardwareInfoVDSCommand, log id: 7308c50c
>2018-05-30 04:55:01,339-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [62e8195a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:55:01,344-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [55274243] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:55:01,349-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2ddb9072] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:55:01,355-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2ddb9072] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 443f0476
>2018-05-30 04:55:01,557-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7c1d3122] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 438d336e
>2018-05-30 04:55:01,559-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7c1d3122] FINISH, GetHardwareInfoVDSCommand, log id: 438d336e
>2018-05-30 04:55:01,571-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [605a8305] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:55:01,576-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [363de69e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:55:01,581-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [7bd033ba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 04:55:01,587-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [7bd033ba] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5616d124
>2018-05-30 04:55:01,738-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ddb9072] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:55:01,738-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2ddb9072] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 443f0476
>2018-05-30 04:55:01,742-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ddb9072] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:55:01,743-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2ddb9072] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:55:01,746-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ddb9072] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:55:01,747-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2ddb9072] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:55:01,751-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ddb9072] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:55:01,752-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2ddb9072] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:55:01,754-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2ddb9072] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:55:01,757-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ddb9072] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2ddb9072, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:55:01,760-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 59fb118
>2018-05-30 04:55:01,760-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:55:01,856-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:55:01,856-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:55:01,856-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:55:01,856-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:55:01,856-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 59fb118
>2018-05-30 04:55:01,856-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:55:01,869-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [56784bbd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:55:01,870-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [56784bbd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 15b0232b
>2018-05-30 04:55:01,872-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [56784bbd] FINISH, SetVdsStatusVDSCommand, log id: 15b0232b
>2018-05-30 04:55:01,878-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [56784bbd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 56784bbd, Job ID: db76f258-a79d-4842-a1d5-3b11b84a1236, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:55:01,882-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [56784bbd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 55274243, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:55:01,884-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [56784bbd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 55274243, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:55:01,896-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [53e755fe] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 04:55:01,897-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [53e755fe] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 04:55:01,953-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7bd033ba] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 04:55:01,953-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [7bd033ba] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5616d124
>2018-05-30 04:55:01,957-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7bd033ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:55:01,957-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7bd033ba] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:55:01,961-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7bd033ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:55:01,962-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7bd033ba] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:55:01,965-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7bd033ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 04:55:01,966-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7bd033ba] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 04:55:01,969-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [7bd033ba] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 04:55:01,972-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7bd033ba] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7bd033ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 04:55:01,975-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2da86659
>2018-05-30 04:55:01,975-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 04:55:02,073-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-33) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:55:02,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 04:55:02,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 04:55:02,073-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 04:55:02,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2da86659
>2018-05-30 04:55:02,073-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-33) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 04:55:02,086-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [72643ff7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:55:02,087-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [72643ff7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: e97bbe6
>2018-05-30 04:55:02,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [72643ff7] FINISH, SetVdsStatusVDSCommand, log id: e97bbe6
>2018-05-30 04:55:02,095-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [72643ff7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 72643ff7, Job ID: f56b61ea-6914-4fdf-b370-ffa9b96d5a56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 04:55:02,099-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [72643ff7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 363de69e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 04:55:02,102-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [72643ff7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 363de69e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 04:55:02,112-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [31290e4e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 04:55:02,113-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [31290e4e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1e6bcfd6] Autorecovering 2 hosts
>2018-05-30 05:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1e6bcfd6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:00:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3a0b167f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:00:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3a0b167f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:00:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3a0b167f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:00:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3a0b167f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:00:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3a0b167f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 69bc1645
>2018-05-30 05:00:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3a0b167f] FINISH, SetVdsStatusVDSCommand, log id: 69bc1645
>2018-05-30 05:00:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3a0b167f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:00:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3a0b167f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:00:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3a0b167f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:00:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [126a0b87] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:00:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [126a0b87] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:00:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [126a0b87] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:00:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [126a0b87] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:00:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [126a0b87] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c38e610
>2018-05-30 05:00:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [126a0b87] FINISH, SetVdsStatusVDSCommand, log id: 2c38e610
>2018-05-30 05:00:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [126a0b87] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6f87dde3
>2018-05-30 05:00:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [126a0b87] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6f87dde3
>2018-05-30 05:00:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [126a0b87] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:00:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [126a0b87] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:00:02,612-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [19d45516] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6cab0603
>2018-05-30 05:00:02,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [19d45516] FINISH, GetHardwareInfoVDSCommand, log id: 6cab0603
>2018-05-30 05:00:02,628-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [5d3ff604] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:00:02,637-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [63a31020] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:00:02,642-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [107226] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:00:02,653-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [107226] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c460ed6
>2018-05-30 05:00:02,763-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [7a4d2cb8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 38fd44be
>2018-05-30 05:00:02,765-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [7a4d2cb8] FINISH, GetHardwareInfoVDSCommand, log id: 38fd44be
>2018-05-30 05:00:02,778-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [585433f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:00:02,782-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [649a25d5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:00:02,787-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1bbe3637] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:00:02,794-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1bbe3637] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 321543a8
>2018-05-30 05:00:03,034-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [107226] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:00:03,034-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [107226] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c460ed6
>2018-05-30 05:00:03,038-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [107226] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:00:03,039-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [107226] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:00:03,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [107226] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:00:03,043-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [107226] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:00:03,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [107226] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:00:03,048-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [107226] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:00:03,051-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [107226] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:00:03,053-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [107226] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 107226, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:00:03,060-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 347d25e6
>2018-05-30 05:00:03,060-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:00:03,154-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:00:03,154-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:00:03,154-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:00:03,154-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:00:03,154-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 347d25e6
>2018-05-30 05:00:03,154-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:00:03,167-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [414bfe4b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:00:03,168-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [414bfe4b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 10f71153
>2018-05-30 05:00:03,171-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [414bfe4b] FINISH, SetVdsStatusVDSCommand, log id: 10f71153
>2018-05-30 05:00:03,203-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [414bfe4b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 414bfe4b, Job ID: 0aef0a8c-2d7a-4e77-b288-c25169f3318d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:00:03,208-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [414bfe4b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 63a31020, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:00:03,210-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [414bfe4b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 63a31020, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:00:03,221-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [3f53093c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:00:03,223-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [3f53093c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:00:05,103-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1bbe3637] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:00:05,103-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1bbe3637] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 321543a8
>2018-05-30 05:00:05,106-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1bbe3637] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:00:05,107-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1bbe3637] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:00:05,111-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1bbe3637] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:00:05,112-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1bbe3637] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:00:05,115-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1bbe3637] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:00:05,116-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1bbe3637] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:00:05,119-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1bbe3637] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:00:05,122-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1bbe3637] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1bbe3637, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:00:05,124-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 55f3ce19
>2018-05-30 05:00:05,125-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:00:05,221-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:00:05,221-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:00:05,221-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:00:05,221-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:00:05,221-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 55f3ce19
>2018-05-30 05:00:05,221-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:00:05,250-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [58fa78d8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:00:05,251-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [58fa78d8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f96cfea
>2018-05-30 05:00:05,254-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [58fa78d8] FINISH, SetVdsStatusVDSCommand, log id: 3f96cfea
>2018-05-30 05:00:05,285-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58fa78d8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 58fa78d8, Job ID: cfef6249-7b4a-4c3c-9da4-5864d57efd27, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:00:05,290-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58fa78d8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 649a25d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:00:05,292-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58fa78d8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 649a25d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:00:05,304-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [7e5eec53] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:00:05,305-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [7e5eec53] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:05:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [61b8e18e] Autorecovering 2 hosts
>2018-05-30 05:05:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [61b8e18e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:05:00,010-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [35feac78] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:05:00,017-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [35feac78] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:05:00,017-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [35feac78] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:05:00,017-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [35feac78] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:05:00,018-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [35feac78] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 63b7c32e
>2018-05-30 05:05:00,020-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [35feac78] FINISH, SetVdsStatusVDSCommand, log id: 63b7c32e
>2018-05-30 05:05:00,021-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [35feac78] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:05:00,023-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [35feac78] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:05:00,023-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [35feac78] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:05:00,028-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [10b4aded] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:05:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [10b4aded] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:05:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [10b4aded] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:05:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [10b4aded] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:05:00,035-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [10b4aded] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2f898978
>2018-05-30 05:05:00,038-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [10b4aded] FINISH, SetVdsStatusVDSCommand, log id: 2f898978
>2018-05-30 05:05:00,040-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [10b4aded] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 21674136
>2018-05-30 05:05:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [10b4aded] FINISH, SetHaMaintenanceModeVDSCommand, log id: 21674136
>2018-05-30 05:05:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [10b4aded] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:05:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [10b4aded] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:05:00,764-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [4af8be81] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 46bf3569
>2018-05-30 05:05:00,766-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [4af8be81] FINISH, GetHardwareInfoVDSCommand, log id: 46bf3569
>2018-05-30 05:05:00,778-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [25ed4f4c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:05:00,786-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [78c02b95] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:05:00,791-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4574092] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:05:00,803-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4574092] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 72272262
>2018-05-30 05:05:01,187-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4574092] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:05:01,187-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4574092] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 72272262
>2018-05-30 05:05:01,191-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4574092] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:05:01,191-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4574092] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:05:01,195-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4574092] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:05:01,196-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4574092] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:05:01,200-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4574092] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:05:01,200-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4574092] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:05:01,203-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4574092] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:05:01,206-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4574092] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4574092, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:05:01,213-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7b151251
>2018-05-30 05:05:01,213-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:05:01,312-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-33) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:05:01,312-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:05:01,312-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:05:01,312-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:05:01,312-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7b151251
>2018-05-30 05:05:01,312-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-33) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:05:01,325-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [3723b44] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:05:01,326-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3723b44] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b5a1b1d
>2018-05-30 05:05:01,329-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3723b44] FINISH, SetVdsStatusVDSCommand, log id: 4b5a1b1d
>2018-05-30 05:05:01,360-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3723b44] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3723b44, Job ID: 6aade07b-ef82-4bbe-99cd-359c425e2b1d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:05:01,365-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3723b44] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 78c02b95, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:05:01,367-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3723b44] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 78c02b95, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:05:01,378-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [3635a245] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:05:01,379-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [3635a245] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:05:03,114-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [68e2db28] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 52d3b119
>2018-05-30 05:05:03,116-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [68e2db28] FINISH, GetHardwareInfoVDSCommand, log id: 52d3b119
>2018-05-30 05:05:03,129-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [72a5e226] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:05:03,133-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [1be2ed6e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:05:03,138-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [23bb10d8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:05:03,145-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [23bb10d8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: da3f1a8
>2018-05-30 05:05:03,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [23bb10d8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:05:03,511-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [23bb10d8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: da3f1a8
>2018-05-30 05:05:03,516-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [23bb10d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:05:03,516-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [23bb10d8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:05:03,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [23bb10d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:05:03,521-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [23bb10d8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:05:03,525-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [23bb10d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:05:03,525-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [23bb10d8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:05:03,528-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [23bb10d8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:05:03,531-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [23bb10d8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 23bb10d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:05:03,533-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3216fc80
>2018-05-30 05:05:03,533-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:05:03,632-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:05:03,632-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:05:03,632-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:05:03,632-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:05:03,632-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3216fc80
>2018-05-30 05:05:03,632-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:05:03,646-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [7a311a4d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:05:03,647-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7a311a4d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4aa8be43
>2018-05-30 05:05:03,649-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7a311a4d] FINISH, SetVdsStatusVDSCommand, log id: 4aa8be43
>2018-05-30 05:05:03,655-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7a311a4d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7a311a4d, Job ID: a7fa7511-46df-43ca-8808-35074a2dcb43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:05:03,659-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7a311a4d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1be2ed6e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:05:03,662-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7a311a4d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1be2ed6e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:05:03,672-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [4825ce17] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:05:03,673-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [4825ce17] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [44a18af8] Autorecovering 2 hosts
>2018-05-30 05:10:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [44a18af8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:10:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [176441ae] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [176441ae] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [176441ae] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:10:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [176441ae] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:10:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [176441ae] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7bddcbb9
>2018-05-30 05:10:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [176441ae] FINISH, SetVdsStatusVDSCommand, log id: 7bddcbb9
>2018-05-30 05:10:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [176441ae] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:10:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [176441ae] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:10:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [176441ae] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:10:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7161c4d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:10:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7161c4d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:10:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7161c4d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:10:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7161c4d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:10:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7161c4d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4cf9fd32
>2018-05-30 05:10:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7161c4d] FINISH, SetVdsStatusVDSCommand, log id: 4cf9fd32
>2018-05-30 05:10:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [7161c4d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 45b0fa9f
>2018-05-30 05:10:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [7161c4d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 45b0fa9f
>2018-05-30 05:10:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7161c4d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:10:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7161c4d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:10:01,349-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3dd2908b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 76731c8
>2018-05-30 05:10:01,351-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3dd2908b] FINISH, GetHardwareInfoVDSCommand, log id: 76731c8
>2018-05-30 05:10:01,364-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7cb58f3d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:10:01,368-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [6fe16ffe] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:10:01,373-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5e64d031] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:10:01,385-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5e64d031] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4ca0886c
>2018-05-30 05:10:01,760-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5e64d031] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:10:01,760-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5e64d031] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4ca0886c
>2018-05-30 05:10:01,764-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5e64d031] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:10:01,765-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5e64d031] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:10:01,769-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5e64d031] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:10:01,770-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5e64d031] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:10:01,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5e64d031] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:10:01,774-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5e64d031] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:10:01,777-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5e64d031] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:10:01,779-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5e64d031] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5e64d031, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:10:01,786-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7fe10a19
>2018-05-30 05:10:01,786-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:10:01,882-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:10:01,882-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:10:01,882-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:10:01,882-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:10:01,882-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7fe10a19
>2018-05-30 05:10:01,882-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:10:01,895-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [7f33dfec] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:10:01,896-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7f33dfec] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 469c7699
>2018-05-30 05:10:01,898-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7f33dfec] FINISH, SetVdsStatusVDSCommand, log id: 469c7699
>2018-05-30 05:10:01,906-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f33dfec] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7f33dfec, Job ID: 35bdb91c-fba2-4f2b-b551-ea80369cd2d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:10:01,910-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f33dfec] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6fe16ffe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:10:01,912-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f33dfec] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6fe16ffe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:10:01,923-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [768c77ad] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:10:01,924-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [768c77ad] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:10:01,947-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4838742c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3bee49df
>2018-05-30 05:10:01,948-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4838742c] FINISH, GetHardwareInfoVDSCommand, log id: 3bee49df
>2018-05-30 05:10:01,960-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [447dfdf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:10:01,964-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [29eddee7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:10:01,969-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [58620569] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:10:01,976-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [58620569] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 12b32aeb
>2018-05-30 05:10:02,345-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58620569] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:10:02,345-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [58620569] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 12b32aeb
>2018-05-30 05:10:02,349-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58620569] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:10:02,349-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [58620569] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:10:02,353-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58620569] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:10:02,354-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [58620569] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:10:02,358-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58620569] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:10:02,359-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [58620569] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:10:02,361-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [58620569] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:10:02,364-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58620569] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 58620569, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:10:02,367-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3c29a9e2
>2018-05-30 05:10:02,367-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:10:02,465-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:10:02,465-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:10:02,465-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:10:02,465-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:10:02,465-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3c29a9e2
>2018-05-30 05:10:02,465-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:10:02,477-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [2ddb9b4e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:10:02,479-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2ddb9b4e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3c490823
>2018-05-30 05:10:02,481-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2ddb9b4e] FINISH, SetVdsStatusVDSCommand, log id: 3c490823
>2018-05-30 05:10:02,486-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2ddb9b4e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2ddb9b4e, Job ID: 36ca2780-1d18-48d8-91af-9e75d091e773, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:10:02,491-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2ddb9b4e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29eddee7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:10:02,493-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2ddb9b4e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29eddee7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:10:02,504-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [5df340e4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:10:02,505-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [5df340e4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7adca4e1] Autorecovering 2 hosts
>2018-05-30 05:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7adca4e1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:15:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7a040cd6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7a040cd6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7a040cd6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:15:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7a040cd6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:15:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7a040cd6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2aff68bd
>2018-05-30 05:15:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7a040cd6] FINISH, SetVdsStatusVDSCommand, log id: 2aff68bd
>2018-05-30 05:15:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7a040cd6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:15:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7a040cd6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:15:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7a040cd6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:15:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5601ae6d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5601ae6d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5601ae6d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:15:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5601ae6d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:15:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5601ae6d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 77d437bc
>2018-05-30 05:15:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5601ae6d] FINISH, SetVdsStatusVDSCommand, log id: 77d437bc
>2018-05-30 05:15:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5601ae6d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4aeaa8aa
>2018-05-30 05:15:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5601ae6d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4aeaa8aa
>2018-05-30 05:15:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5601ae6d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:15:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5601ae6d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:15:02,632-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [75743d53] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2d66d61d
>2018-05-30 05:15:02,634-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [75743d53] FINISH, GetHardwareInfoVDSCommand, log id: 2d66d61d
>2018-05-30 05:15:02,647-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [5fb7cd74] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:15:02,655-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [483ed0e0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:15:02,661-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [54d14798] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:15:02,672-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [54d14798] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 721752fd
>2018-05-30 05:15:03,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54d14798] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:15:03,047-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [54d14798] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 721752fd
>2018-05-30 05:15:03,051-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54d14798] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:15:03,052-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [54d14798] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:15:03,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54d14798] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:15:03,056-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [54d14798] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:15:03,060-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54d14798] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:15:03,061-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [54d14798] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:15:03,064-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [54d14798] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:15:03,066-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54d14798] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 54d14798, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:15:03,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7f5ab061
>2018-05-30 05:15:03,073-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:15:03,120-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [79b3ec92] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 71dc0de5
>2018-05-30 05:15:03,122-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [79b3ec92] FINISH, GetHardwareInfoVDSCommand, log id: 71dc0de5
>2018-05-30 05:15:03,134-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6a223e88] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:15:03,139-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [1d7ef558] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:15:03,144-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7959623d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:15:03,151-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7959623d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 63730f89
>2018-05-30 05:15:03,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:15:03,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:15:03,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:15:03,178-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:15:03,178-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7f5ab061
>2018-05-30 05:15:03,178-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:15:03,192-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5c58d55c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:15:03,193-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5c58d55c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4cb913d5
>2018-05-30 05:15:03,195-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5c58d55c] FINISH, SetVdsStatusVDSCommand, log id: 4cb913d5
>2018-05-30 05:15:03,227-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5c58d55c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5c58d55c, Job ID: 638972ee-8806-458e-a845-62cb55f34233, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:15:03,232-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5c58d55c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 483ed0e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:15:03,234-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5c58d55c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 483ed0e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:15:03,245-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [f26fca3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:15:03,246-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [f26fca3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:15:03,529-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7959623d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:15:03,529-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7959623d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 63730f89
>2018-05-30 05:15:03,533-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7959623d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:15:03,533-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7959623d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:15:03,537-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7959623d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:15:03,538-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7959623d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:15:03,542-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7959623d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:15:03,543-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7959623d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:15:03,545-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7959623d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:15:03,548-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7959623d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7959623d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:15:03,551-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6604b128
>2018-05-30 05:15:03,551-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:15:03,648-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:15:03,648-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:15:03,648-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:15:03,648-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:15:03,648-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6604b128
>2018-05-30 05:15:03,649-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:15:03,661-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [593f548a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:15:03,662-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [593f548a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7e9fdc93
>2018-05-30 05:15:03,664-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [593f548a] FINISH, SetVdsStatusVDSCommand, log id: 7e9fdc93
>2018-05-30 05:15:03,670-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [593f548a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 593f548a, Job ID: 0a157851-8341-4a21-b2fb-730e0d297f3a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:15:03,675-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [593f548a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d7ef558, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:15:03,677-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [593f548a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d7ef558, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:15:03,687-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [7e2dee54] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:15:03,688-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [7e2dee54] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:20:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5935ee77] Autorecovering 2 hosts
>2018-05-30 05:20:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5935ee77] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:20:00,012-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f71786] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:20:00,019-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f71786] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:20:00,019-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f71786] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:20:00,019-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f71786] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:20:00,021-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [f71786] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 71086562
>2018-05-30 05:20:00,023-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [f71786] FINISH, SetVdsStatusVDSCommand, log id: 71086562
>2018-05-30 05:20:00,025-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f71786] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:20:00,026-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f71786] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:20:00,026-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [f71786] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:20:00,032-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3707aee8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:20:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3707aee8] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:20:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3707aee8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:20:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3707aee8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:20:00,039-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3707aee8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2da8f045
>2018-05-30 05:20:00,041-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3707aee8] FINISH, SetVdsStatusVDSCommand, log id: 2da8f045
>2018-05-30 05:20:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3707aee8] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3c046754
>2018-05-30 05:20:00,046-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3707aee8] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3c046754
>2018-05-30 05:20:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3707aee8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:20:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3707aee8] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:20:00,888-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6add5587] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 585b6231
>2018-05-30 05:20:00,890-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6add5587] FINISH, GetHardwareInfoVDSCommand, log id: 585b6231
>2018-05-30 05:20:00,903-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [4c39db13] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:20:00,907-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3821f8f3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:20:00,912-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [45274358] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:20:00,919-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [45274358] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 250d8ff1
>2018-05-30 05:20:01,261-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [456037d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 41fea7d5
>2018-05-30 05:20:01,262-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [456037d] FINISH, GetHardwareInfoVDSCommand, log id: 41fea7d5
>2018-05-30 05:20:01,275-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7839b6e7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:20:01,279-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [495c6d7e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:20:01,285-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1dcda842] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:20:01,289-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [45274358] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:20:01,290-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [45274358] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 250d8ff1
>2018-05-30 05:20:01,292-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1dcda842] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 41356f43
>2018-05-30 05:20:01,294-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [45274358] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:20:01,294-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [45274358] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:20:01,298-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [45274358] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:20:01,299-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [45274358] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:20:01,302-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [45274358] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:20:01,303-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [45274358] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:20:01,306-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [45274358] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:20:01,309-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [45274358] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 45274358, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:20:01,316-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 117f218e
>2018-05-30 05:20:01,316-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:20:01,414-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:20:01,414-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:20:01,414-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:20:01,414-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:20:01,414-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 117f218e
>2018-05-30 05:20:01,414-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:20:01,426-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [76f4520c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:20:01,428-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [76f4520c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c7addbe
>2018-05-30 05:20:01,430-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [76f4520c] FINISH, SetVdsStatusVDSCommand, log id: 7c7addbe
>2018-05-30 05:20:01,462-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [76f4520c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 76f4520c, Job ID: 8524496c-09ee-44af-ab0f-6596462b9b89, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:20:01,467-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [76f4520c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3821f8f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:20:01,469-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [76f4520c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3821f8f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:20:01,480-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [38e395ac] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:20:01,481-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [38e395ac] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:20:01,676-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1dcda842] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:20:01,676-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1dcda842] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 41356f43
>2018-05-30 05:20:01,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1dcda842] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:20:01,681-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1dcda842] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:20:01,685-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1dcda842] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:20:01,686-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1dcda842] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:20:01,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1dcda842] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:20:01,691-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1dcda842] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:20:01,694-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1dcda842] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:20:01,697-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1dcda842] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1dcda842, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:20:01,700-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4d289cbd
>2018-05-30 05:20:01,700-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:20:01,797-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:20:01,797-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:20:01,797-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:20:01,797-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:20:01,797-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4d289cbd
>2018-05-30 05:20:01,797-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:20:01,810-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [55f40f20] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:20:01,811-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [55f40f20] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 93b5b9a
>2018-05-30 05:20:01,813-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [55f40f20] FINISH, SetVdsStatusVDSCommand, log id: 93b5b9a
>2018-05-30 05:20:01,818-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [55f40f20] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 55f40f20, Job ID: 1f71179a-1049-4469-b4c8-593d54cd8dd2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:20:01,823-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [55f40f20] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 495c6d7e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:20:01,825-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [55f40f20] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 495c6d7e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:20:01,836-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [75490a9d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:20:01,837-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [75490a9d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [72dad077] Autorecovering 2 hosts
>2018-05-30 05:25:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [72dad077] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:25:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [70a2f4e3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:25:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [70a2f4e3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:25:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [70a2f4e3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:25:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [70a2f4e3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:25:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [70a2f4e3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72e488c4
>2018-05-30 05:25:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [70a2f4e3] FINISH, SetVdsStatusVDSCommand, log id: 72e488c4
>2018-05-30 05:25:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [70a2f4e3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:25:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [70a2f4e3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:25:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [70a2f4e3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:25:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5345fc19] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5345fc19] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5345fc19] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:25:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5345fc19] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:25:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5345fc19] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6e02b7f9
>2018-05-30 05:25:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5345fc19] FINISH, SetVdsStatusVDSCommand, log id: 6e02b7f9
>2018-05-30 05:25:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5345fc19] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 51107af4
>2018-05-30 05:25:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5345fc19] FINISH, SetHaMaintenanceModeVDSCommand, log id: 51107af4
>2018-05-30 05:25:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5345fc19] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:25:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5345fc19] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:25:02,182-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5811269b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1d9a5669
>2018-05-30 05:25:02,184-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5811269b] FINISH, GetHardwareInfoVDSCommand, log id: 1d9a5669
>2018-05-30 05:25:02,197-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [638b9564] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:25:02,205-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [4d3569c1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:25:02,210-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3031b5b5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:25:02,264-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3031b5b5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 74efdaf2
>2018-05-30 05:25:02,395-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1eee0dbb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 48e22591
>2018-05-30 05:25:02,397-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1eee0dbb] FINISH, GetHardwareInfoVDSCommand, log id: 48e22591
>2018-05-30 05:25:02,410-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [205c2c71] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:25:02,415-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [73622248] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:25:02,420-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [29f1293d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:25:02,426-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [29f1293d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4de447bf
>2018-05-30 05:25:02,641-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3031b5b5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:25:02,641-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3031b5b5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 74efdaf2
>2018-05-30 05:25:02,645-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3031b5b5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:25:02,646-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3031b5b5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:25:02,649-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3031b5b5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:25:02,650-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3031b5b5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:25:02,653-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3031b5b5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:25:02,654-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3031b5b5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:25:02,657-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3031b5b5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:25:02,659-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3031b5b5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3031b5b5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:25:02,666-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 68cf6cb1
>2018-05-30 05:25:02,666-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:25:02,765-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:25:02,765-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:25:02,765-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:25:02,765-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:25:02,765-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 68cf6cb1
>2018-05-30 05:25:02,765-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:25:02,778-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [61543945] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:25:02,779-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [61543945] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 23c86e77
>2018-05-30 05:25:02,782-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [61543945] FINISH, SetVdsStatusVDSCommand, log id: 23c86e77
>2018-05-30 05:25:02,800-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29f1293d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:25:02,800-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [29f1293d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4de447bf
>2018-05-30 05:25:02,805-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29f1293d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:25:02,806-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [29f1293d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:25:02,810-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29f1293d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:25:02,810-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [29f1293d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:25:02,814-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29f1293d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:25:02,815-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [29f1293d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:25:02,816-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [61543945] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 61543945, Job ID: 0c5fd6e7-22f5-41ab-b13a-18e1c0bda662, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:25:02,819-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [29f1293d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:25:02,821-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [61543945] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4d3569c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:25:02,821-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29f1293d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 29f1293d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:25:02,823-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [61543945] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4d3569c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:25:02,825-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 64e85e38
>2018-05-30 05:25:02,825-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:25:02,835-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [b46e17] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:25:02,836-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [b46e17] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:25:02,921-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:25:02,921-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:25:02,921-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:25:02,921-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:25:02,921-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 64e85e38
>2018-05-30 05:25:02,921-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:25:02,935-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [159b380] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:25:02,936-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [159b380] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 573a02d1
>2018-05-30 05:25:02,939-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [159b380] FINISH, SetVdsStatusVDSCommand, log id: 573a02d1
>2018-05-30 05:25:02,970-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [159b380] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 159b380, Job ID: 69b069e2-f587-41e4-8e70-08d43f4e5e29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:25:02,975-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [159b380] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 73622248, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:25:02,977-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [159b380] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 73622248, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:25:02,989-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [4f6dc3ad] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:25:02,990-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [4f6dc3ad] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [50dbb572] Autorecovering 2 hosts
>2018-05-30 05:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [50dbb572] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:30:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [359478ac] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [359478ac] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [359478ac] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:30:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [359478ac] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:30:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [359478ac] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 764e2486
>2018-05-30 05:30:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [359478ac] FINISH, SetVdsStatusVDSCommand, log id: 764e2486
>2018-05-30 05:30:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [359478ac] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:30:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [359478ac] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:30:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [359478ac] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:30:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4036db81] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:30:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4036db81] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:30:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4036db81] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:30:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4036db81] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:30:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4036db81] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54e290e0
>2018-05-30 05:30:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4036db81] FINISH, SetVdsStatusVDSCommand, log id: 54e290e0
>2018-05-30 05:30:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4036db81] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 729bc049
>2018-05-30 05:30:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4036db81] FINISH, SetHaMaintenanceModeVDSCommand, log id: 729bc049
>2018-05-30 05:30:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4036db81] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:30:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4036db81] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:30:00,540-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [30eba73f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4b25a939
>2018-05-30 05:30:00,541-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [30eba73f] FINISH, GetHardwareInfoVDSCommand, log id: 4b25a939
>2018-05-30 05:30:00,569-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [60099d65] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:30:00,572-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [43cc5a80] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 67a27a9c
>2018-05-30 05:30:00,573-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [43cc5a80] FINISH, GetHardwareInfoVDSCommand, log id: 67a27a9c
>2018-05-30 05:30:00,578-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [400ccc10] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:30:00,584-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [133b5a7d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:30:00,587-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [395ddc4b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:30:00,596-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [d239ab8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:30:00,597-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [133b5a7d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3160fe05
>2018-05-30 05:30:00,601-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1890df75] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:30:00,613-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1890df75] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 36f642be
>2018-05-30 05:30:01,975-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [133b5a7d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:30:01,975-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [133b5a7d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3160fe05
>2018-05-30 05:30:01,979-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [133b5a7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:30:01,980-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [133b5a7d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:30:01,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1890df75] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:30:01,983-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1890df75] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 36f642be
>2018-05-30 05:30:01,984-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [133b5a7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:30:01,985-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [133b5a7d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:30:01,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1890df75] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:30:01,988-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1890df75] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:30:01,989-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [133b5a7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:30:01,990-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [133b5a7d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:30:01,991-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1890df75] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:30:01,992-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1890df75] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:30:01,993-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [133b5a7d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:30:01,995-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [133b5a7d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 133b5a7d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:30:01,996-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1890df75] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:30:01,997-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1890df75] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:30:02,000-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1890df75] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:30:02,002-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1890df75] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1890df75, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:30:02,003-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3808f16a
>2018-05-30 05:30:02,003-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:30:02,100-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:30:02,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:30:02,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:30:02,100-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:30:02,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3808f16a
>2018-05-30 05:30:02,100-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:30:02,106-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 545293e3
>2018-05-30 05:30:02,106-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:30:02,113-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [5ff70546] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:30:02,114-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5ff70546] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5719fc29
>2018-05-30 05:30:02,116-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5ff70546] FINISH, SetVdsStatusVDSCommand, log id: 5719fc29
>2018-05-30 05:30:02,146-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5ff70546] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5ff70546, Job ID: 7524560f-7f81-457b-b27b-fd432a8dba23, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:30:02,151-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5ff70546] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 400ccc10, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:30:02,153-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5ff70546] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 400ccc10, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:30:02,163-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [58bc5a3d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:30:02,164-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [58bc5a3d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:30:02,205-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:30:02,205-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:30:02,205-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:30:02,205-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:30:02,205-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 545293e3
>2018-05-30 05:30:02,205-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:30:02,218-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [5091cc6c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:30:02,219-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5091cc6c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1ac92743
>2018-05-30 05:30:02,221-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5091cc6c] FINISH, SetVdsStatusVDSCommand, log id: 1ac92743
>2018-05-30 05:30:02,226-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5091cc6c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5091cc6c, Job ID: acbef4f0-6d30-430d-bf48-a9965f9f2e00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:30:02,231-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5091cc6c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: d239ab8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:30:02,233-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5091cc6c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: d239ab8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:30:02,243-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [31c81795] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:30:02,244-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [31c81795] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:35:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [41241d02] Autorecovering 2 hosts
>2018-05-30 05:35:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [41241d02] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:35:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1951a101] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:35:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1951a101] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:35:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1951a101] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:35:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1951a101] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:35:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1951a101] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1a22d284
>2018-05-30 05:35:00,045-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1951a101] FINISH, SetVdsStatusVDSCommand, log id: 1a22d284
>2018-05-30 05:35:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1951a101] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:35:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1951a101] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:35:00,047-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [1951a101] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:35:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5ea6738e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:35:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5ea6738e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:35:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5ea6738e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:35:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5ea6738e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:35:00,060-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5ea6738e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7e24cbd9
>2018-05-30 05:35:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5ea6738e] FINISH, SetVdsStatusVDSCommand, log id: 7e24cbd9
>2018-05-30 05:35:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [5ea6738e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: d5db332
>2018-05-30 05:35:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [5ea6738e] FINISH, SetHaMaintenanceModeVDSCommand, log id: d5db332
>2018-05-30 05:35:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5ea6738e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:35:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5ea6738e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:35:02,765-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [50ea5884] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2cad5603
>2018-05-30 05:35:02,767-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [50ea5884] FINISH, GetHardwareInfoVDSCommand, log id: 2cad5603
>2018-05-30 05:35:02,780-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [14e1da4c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:35:02,785-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [13a7225e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:35:02,790-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [281ffc46] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:35:02,802-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [281ffc46] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 29b1b75f
>2018-05-30 05:35:02,937-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6323ba6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7fc13d47
>2018-05-30 05:35:02,939-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6323ba6] FINISH, GetHardwareInfoVDSCommand, log id: 7fc13d47
>2018-05-30 05:35:02,951-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3f81543c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:35:02,955-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [3551fb78] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:35:02,960-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [bac7355] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:35:03,022-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [bac7355] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5361c6a
>2018-05-30 05:35:03,182-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [281ffc46] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:35:03,182-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [281ffc46] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 29b1b75f
>2018-05-30 05:35:03,186-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [281ffc46] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:35:03,187-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [281ffc46] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:35:03,191-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [281ffc46] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:35:03,191-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [281ffc46] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:35:03,195-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [281ffc46] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:35:03,196-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [281ffc46] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:35:03,199-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [281ffc46] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:35:03,202-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [281ffc46] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 281ffc46, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:35:03,208-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 22a87390
>2018-05-30 05:35:03,208-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:35:03,305-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-33) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:35:03,305-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:35:03,305-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:35:03,305-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:35:03,305-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] FINISH, ConnectStoragePoolVDSCommand, log id: 22a87390
>2018-05-30 05:35:03,305-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-33) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:35:03,319-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [5684fc99] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:35:03,320-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5684fc99] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 22d5e850
>2018-05-30 05:35:03,322-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5684fc99] FINISH, SetVdsStatusVDSCommand, log id: 22d5e850
>2018-05-30 05:35:03,330-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5684fc99] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5684fc99, Job ID: ec745064-bbc0-4023-88b8-dc31f2943056, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:35:03,334-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5684fc99] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 13a7225e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:35:03,336-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5684fc99] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 13a7225e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:35:03,347-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [7177edb0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:35:03,348-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [7177edb0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:35:03,394-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bac7355] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:35:03,394-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [bac7355] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5361c6a
>2018-05-30 05:35:03,398-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bac7355] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:35:03,399-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [bac7355] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:35:03,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bac7355] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:35:03,404-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [bac7355] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:35:03,407-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bac7355] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:35:03,408-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [bac7355] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:35:03,411-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [bac7355] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:35:03,414-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bac7355] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: bac7355, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:35:03,417-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6ddf9e1f
>2018-05-30 05:35:03,417-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:35:03,538-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:35:03,538-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:35:03,538-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:35:03,538-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:35:03,538-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6ddf9e1f
>2018-05-30 05:35:03,538-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:35:03,552-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [78d6fd3f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:35:03,553-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [78d6fd3f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 60e97791
>2018-05-30 05:35:03,555-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [78d6fd3f] FINISH, SetVdsStatusVDSCommand, log id: 60e97791
>2018-05-30 05:35:03,561-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78d6fd3f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 78d6fd3f, Job ID: 802b1af5-903f-476a-8e8e-a8a5d2eb7c35, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:35:03,565-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78d6fd3f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3551fb78, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:35:03,567-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78d6fd3f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3551fb78, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:35:03,578-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [21caba26] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:35:03,579-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [21caba26] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:40:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5232af6b] Autorecovering 2 hosts
>2018-05-30 05:40:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5232af6b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:40:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68cde0e1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68cde0e1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68cde0e1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:40:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68cde0e1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:40:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [68cde0e1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c2ad78d
>2018-05-30 05:40:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [68cde0e1] FINISH, SetVdsStatusVDSCommand, log id: 7c2ad78d
>2018-05-30 05:40:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68cde0e1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:40:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68cde0e1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:40:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [68cde0e1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:40:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [47922f50] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [47922f50] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [47922f50] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [47922f50] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:40:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [47922f50] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 75010a41
>2018-05-30 05:40:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [47922f50] FINISH, SetVdsStatusVDSCommand, log id: 75010a41
>2018-05-30 05:40:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [47922f50] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 49af71ce
>2018-05-30 05:40:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [47922f50] FINISH, SetHaMaintenanceModeVDSCommand, log id: 49af71ce
>2018-05-30 05:40:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [47922f50] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:40:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [47922f50] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:40:01,015-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [72f47282] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 74856397
>2018-05-30 05:40:01,016-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [72f47282] FINISH, GetHardwareInfoVDSCommand, log id: 74856397
>2018-05-30 05:40:01,029-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [7589a5c0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:40:01,038-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [7ccd8632] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:40:01,043-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2fdbde0b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:40:01,054-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2fdbde0b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 786a5a50
>2018-05-30 05:40:01,244-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [27a83f1d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 794a6d3b
>2018-05-30 05:40:01,246-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [27a83f1d] FINISH, GetHardwareInfoVDSCommand, log id: 794a6d3b
>2018-05-30 05:40:01,258-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [2eae53fa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:40:01,262-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [2fe1ddbc] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:40:01,267-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [515f68f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:40:01,274-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [515f68f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: ecb2c3c
>2018-05-30 05:40:01,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fdbde0b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:40:01,432-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2fdbde0b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 786a5a50
>2018-05-30 05:40:01,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fdbde0b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:40:01,437-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2fdbde0b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:40:01,440-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fdbde0b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:40:01,441-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2fdbde0b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:40:01,444-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fdbde0b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:40:01,445-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2fdbde0b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:40:01,447-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2fdbde0b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:40:01,450-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fdbde0b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2fdbde0b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:40:01,458-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 395f80fc
>2018-05-30 05:40:01,458-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:40:01,554-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:40:01,554-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:40:01,554-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:40:01,554-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:40:01,554-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 395f80fc
>2018-05-30 05:40:01,554-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:40:01,567-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [2efcd448] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:40:01,568-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2efcd448] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4822c41e
>2018-05-30 05:40:01,570-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2efcd448] FINISH, SetVdsStatusVDSCommand, log id: 4822c41e
>2018-05-30 05:40:01,602-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2efcd448] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2efcd448, Job ID: a5e0e29b-ace9-436b-aaf0-12989588961c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:40:01,607-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2efcd448] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7ccd8632, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:40:01,609-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2efcd448] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7ccd8632, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:40:01,619-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [36c5cf5e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:40:01,620-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [36c5cf5e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:40:01,647-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [515f68f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:40:01,647-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [515f68f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: ecb2c3c
>2018-05-30 05:40:01,650-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [515f68f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:40:01,651-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [515f68f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:40:01,655-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [515f68f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:40:01,656-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [515f68f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:40:01,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [515f68f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:40:01,660-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [515f68f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:40:01,663-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [515f68f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:40:01,666-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [515f68f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 515f68f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:40:01,668-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 35dca898
>2018-05-30 05:40:01,668-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:40:01,765-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:40:01,765-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:40:01,765-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:40:01,765-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:40:01,765-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 35dca898
>2018-05-30 05:40:01,765-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:40:01,778-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [18b18a57] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:40:01,779-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [18b18a57] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2eceaa6
>2018-05-30 05:40:01,781-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [18b18a57] FINISH, SetVdsStatusVDSCommand, log id: 2eceaa6
>2018-05-30 05:40:01,786-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18b18a57] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 18b18a57, Job ID: 3084ecd0-6c89-4c02-b6c8-e25b19e1a361, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:40:01,791-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18b18a57] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2fe1ddbc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:40:01,793-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18b18a57] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2fe1ddbc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:40:01,803-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [1e081033] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:40:01,804-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [1e081033] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6ace51fa] Autorecovering 2 hosts
>2018-05-30 05:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6ace51fa] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:45:00,036-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a19688] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:45:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a19688] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:45:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a19688] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:45:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a19688] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:45:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2a19688] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 78b9db37
>2018-05-30 05:45:00,047-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2a19688] FINISH, SetVdsStatusVDSCommand, log id: 78b9db37
>2018-05-30 05:45:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a19688] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:45:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a19688] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:45:00,050-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2a19688] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:45:00,056-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2bad4c36] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:45:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2bad4c36] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:45:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2bad4c36] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:45:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2bad4c36] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:45:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2bad4c36] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 556b98f9
>2018-05-30 05:45:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2bad4c36] FINISH, SetVdsStatusVDSCommand, log id: 556b98f9
>2018-05-30 05:45:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2bad4c36] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 259ea70c
>2018-05-30 05:45:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2bad4c36] FINISH, SetHaMaintenanceModeVDSCommand, log id: 259ea70c
>2018-05-30 05:45:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2bad4c36] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:45:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2bad4c36] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:45:02,200-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [658d8ca8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6667a756
>2018-05-30 05:45:02,202-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [658d8ca8] FINISH, GetHardwareInfoVDSCommand, log id: 6667a756
>2018-05-30 05:45:02,215-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [67765d21] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:45:02,223-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [5e315b00] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:45:02,228-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [497c1251] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:45:02,240-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [497c1251] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2ebba05f
>2018-05-30 05:45:02,625-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [497c1251] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:45:02,625-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [497c1251] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2ebba05f
>2018-05-30 05:45:02,629-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [497c1251] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:45:02,629-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [497c1251] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:45:02,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [497c1251] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:45:02,634-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [497c1251] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:45:02,638-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [497c1251] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:45:02,638-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [497c1251] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:45:02,641-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [497c1251] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:45:02,644-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [497c1251] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 497c1251, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:45:02,650-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5935c0e8
>2018-05-30 05:45:02,650-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:45:02,725-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [79e08d18] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 687638e7
>2018-05-30 05:45:02,727-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [79e08d18] FINISH, GetHardwareInfoVDSCommand, log id: 687638e7
>2018-05-30 05:45:02,740-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [3d13a5b2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:45:02,744-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [85fc816] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:45:02,749-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:45:02,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:45:02,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:45:02,749-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:45:02,749-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5935c0e8
>2018-05-30 05:45:02,749-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:45:02,750-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [559dae05] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:45:02,758-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [559dae05] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 55bba857
>2018-05-30 05:45:02,766-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [2ff8ddeb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:45:02,767-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2ff8ddeb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b78d972
>2018-05-30 05:45:02,769-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2ff8ddeb] FINISH, SetVdsStatusVDSCommand, log id: 4b78d972
>2018-05-30 05:45:02,777-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2ff8ddeb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2ff8ddeb, Job ID: 97fcb827-ec8f-4058-953f-909ea4265774, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:45:02,782-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2ff8ddeb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5e315b00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:45:02,784-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2ff8ddeb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5e315b00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:45:02,795-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [6880039c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:45:02,796-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [6880039c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:45:03,128-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [559dae05] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:45:03,129-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [559dae05] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 55bba857
>2018-05-30 05:45:03,132-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [559dae05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:45:03,133-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [559dae05] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:45:03,137-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [559dae05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:45:03,137-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [559dae05] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:45:03,141-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [559dae05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:45:03,142-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [559dae05] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:45:03,145-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [559dae05] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:45:03,148-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [559dae05] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 559dae05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:45:03,151-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3fa3a718
>2018-05-30 05:45:03,151-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:45:03,250-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:45:03,250-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:45:03,250-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:45:03,250-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:45:03,250-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3fa3a718
>2018-05-30 05:45:03,250-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:45:03,263-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [7878061c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:45:03,264-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7878061c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 81eaa92
>2018-05-30 05:45:03,267-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7878061c] FINISH, SetVdsStatusVDSCommand, log id: 81eaa92
>2018-05-30 05:45:03,272-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7878061c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7878061c, Job ID: d72886de-c8da-49a9-9d2a-a1bed3ada8e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:45:03,277-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7878061c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 85fc816, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:45:03,280-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7878061c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 85fc816, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:45:03,290-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2de7d0f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:45:03,291-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2de7d0f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [3fbf2294] Autorecovering 2 hosts
>2018-05-30 05:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [3fbf2294] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:50:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70a6a9aa] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:50:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70a6a9aa] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:50:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70a6a9aa] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:50:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70a6a9aa] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:50:00,046-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [70a6a9aa] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 28510106
>2018-05-30 05:50:00,049-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [70a6a9aa] FINISH, SetVdsStatusVDSCommand, log id: 28510106
>2018-05-30 05:50:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70a6a9aa] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:50:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70a6a9aa] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:50:00,052-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [70a6a9aa] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:50:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [263e0d76] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:50:00,111-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [263e0d76] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:50:00,111-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [263e0d76] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:50:00,111-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [263e0d76] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:50:00,113-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [263e0d76] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 27f0633c
>2018-05-30 05:50:00,116-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [263e0d76] FINISH, SetVdsStatusVDSCommand, log id: 27f0633c
>2018-05-30 05:50:00,119-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [263e0d76] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6ff54187
>2018-05-30 05:50:00,121-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [263e0d76] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6ff54187
>2018-05-30 05:50:00,121-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [263e0d76] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:50:00,122-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [263e0d76] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:50:00,406-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2baa4e5f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5b381780
>2018-05-30 05:50:00,408-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2baa4e5f] FINISH, GetHardwareInfoVDSCommand, log id: 5b381780
>2018-05-30 05:50:00,420-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6a8265ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:50:00,428-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [632a55a0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:50:00,433-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1a9c30c1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:50:00,496-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1a9c30c1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3b1df337
>2018-05-30 05:50:00,877-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a9c30c1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:50:00,877-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1a9c30c1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3b1df337
>2018-05-30 05:50:00,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a9c30c1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:50:00,881-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1a9c30c1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:50:00,885-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a9c30c1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:50:00,885-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1a9c30c1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:50:00,889-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a9c30c1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:50:00,890-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1a9c30c1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:50:00,893-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1a9c30c1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:50:00,896-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a9c30c1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1a9c30c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:50:00,903-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 792b848c
>2018-05-30 05:50:00,903-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:50:00,907-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [11c900ad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 69a048fe
>2018-05-30 05:50:00,909-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [11c900ad] FINISH, GetHardwareInfoVDSCommand, log id: 69a048fe
>2018-05-30 05:50:00,921-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [f86259b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:50:00,926-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [4bd2fd65] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:50:00,931-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6c6687fb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:50:00,938-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6c6687fb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5365bc45
>2018-05-30 05:50:01,003-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:50:01,003-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:50:01,003-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:50:01,003-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:50:01,003-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 792b848c
>2018-05-30 05:50:01,003-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:50:01,015-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [2e2616bf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:50:01,017-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2e2616bf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: cdce96e
>2018-05-30 05:50:01,019-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2e2616bf] FINISH, SetVdsStatusVDSCommand, log id: cdce96e
>2018-05-30 05:50:01,027-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2e2616bf] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2e2616bf, Job ID: 4430a226-d629-4502-a591-b4b17da7d57c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:50:01,031-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2e2616bf] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 632a55a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:50:01,033-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2e2616bf] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 632a55a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:50:01,044-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [d4e21b2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:50:01,045-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [d4e21b2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:50:01,309-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6c6687fb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:50:01,309-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6c6687fb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5365bc45
>2018-05-30 05:50:01,313-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6c6687fb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:50:01,314-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6c6687fb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:50:01,317-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6c6687fb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:50:01,318-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6c6687fb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:50:01,322-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6c6687fb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:50:01,322-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6c6687fb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:50:01,325-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6c6687fb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:50:01,328-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6c6687fb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6c6687fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:50:01,334-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4b9870ca
>2018-05-30 05:50:01,334-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:50:01,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:50:01,432-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:50:01,432-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:50:01,432-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:50:01,433-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4b9870ca
>2018-05-30 05:50:01,433-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:50:01,445-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [1c5a5592] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:50:01,446-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1c5a5592] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5eef7a2a
>2018-05-30 05:50:01,448-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1c5a5592] FINISH, SetVdsStatusVDSCommand, log id: 5eef7a2a
>2018-05-30 05:50:01,453-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1c5a5592] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1c5a5592, Job ID: 06004afd-c82a-47c6-9d0a-4df8745caf68, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:50:01,458-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1c5a5592] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4bd2fd65, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:50:01,460-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1c5a5592] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4bd2fd65, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:50:01,470-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [79e915f2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:50:01,471-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [79e915f2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:54:43,557-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler9) [64a61388] Attempting to update VMs/Templates Ovf.
>2018-05-30 05:54:43,560-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 05:54:43,560-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 05:54:43,560-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:54:43,565-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-30 05:54:43,568-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-30 05:54:43,568-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Attempting to update template OVFs in Data Center 'Default'
>2018-05-30 05:54:43,568-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-30 05:54:43,568-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 05:54:43,569-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 05:54:43,569-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 05:55:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [be79151] Autorecovering 2 hosts
>2018-05-30 05:55:00,010-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [be79151] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:55:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [154a736c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:55:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [154a736c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:55:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [154a736c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:55:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [154a736c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:55:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [154a736c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3672a3c5
>2018-05-30 05:55:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [154a736c] FINISH, SetVdsStatusVDSCommand, log id: 3672a3c5
>2018-05-30 05:55:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [154a736c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:55:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [154a736c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 05:55:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [154a736c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:55:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9769b2b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:55:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9769b2b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 05:55:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9769b2b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:55:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9769b2b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:55:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [9769b2b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b2afeb
>2018-05-30 05:55:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [9769b2b] FINISH, SetVdsStatusVDSCommand, log id: 3b2afeb
>2018-05-30 05:55:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [9769b2b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4a94744
>2018-05-30 05:55:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [9769b2b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4a94744
>2018-05-30 05:55:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9769b2b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 05:55:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9769b2b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 05:55:01,664-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3673234f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 11d13e99
>2018-05-30 05:55:01,665-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3673234f] FINISH, GetHardwareInfoVDSCommand, log id: 11d13e99
>2018-05-30 05:55:01,678-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [75e7aff2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:55:01,687-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [7e0a96ed] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:55:01,692-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [d9dbf5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:55:01,703-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [d9dbf5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3f0206b4
>2018-05-30 05:55:02,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [d9dbf5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:55:02,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [d9dbf5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3f0206b4
>2018-05-30 05:55:02,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [d9dbf5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:55:02,081-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [d9dbf5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:55:02,085-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [d9dbf5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:55:02,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [d9dbf5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:55:02,089-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [d9dbf5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:55:02,090-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [d9dbf5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:55:02,093-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [d9dbf5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:55:02,096-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [d9dbf5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: d9dbf5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:55:02,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5d0304db
>2018-05-30 05:55:02,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:55:02,139-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6d930ea] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6a1e0881
>2018-05-30 05:55:02,141-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6d930ea] FINISH, GetHardwareInfoVDSCommand, log id: 6a1e0881
>2018-05-30 05:55:02,153-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [7355d217] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:55:02,157-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [340727ea] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:55:02,162-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6386a71e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 05:55:02,168-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6386a71e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 308848fe
>2018-05-30 05:55:02,197-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:55:02,197-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:55:02,197-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:55:02,197-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:55:02,197-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5d0304db
>2018-05-30 05:55:02,197-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:55:02,210-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [16e29c17] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:55:02,212-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [16e29c17] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43c989a
>2018-05-30 05:55:02,214-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [16e29c17] FINISH, SetVdsStatusVDSCommand, log id: 43c989a
>2018-05-30 05:55:02,245-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [16e29c17] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 16e29c17, Job ID: 5d6acdb3-e78e-409b-8487-e9b6f7005232, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:55:02,250-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [16e29c17] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7e0a96ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:55:02,252-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [16e29c17] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7e0a96ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:55:02,263-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [2e87dc70] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 05:55:02,264-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [2e87dc70] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 05:55:02,534-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6386a71e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 05:55:02,534-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6386a71e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 308848fe
>2018-05-30 05:55:02,538-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6386a71e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:55:02,539-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6386a71e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:55:02,543-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6386a71e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:55:02,543-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6386a71e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:55:02,547-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6386a71e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 05:55:02,548-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6386a71e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 05:55:02,551-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6386a71e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 05:55:02,554-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6386a71e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6386a71e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 05:55:02,557-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 546973f
>2018-05-30 05:55:02,557-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 05:55:02,652-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:55:02,652-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 05:55:02,652-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 05:55:02,652-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 05:55:02,652-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 546973f
>2018-05-30 05:55:02,652-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 05:55:02,666-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [72c6a14e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:55:02,667-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [72c6a14e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: cefb307
>2018-05-30 05:55:02,669-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [72c6a14e] FINISH, SetVdsStatusVDSCommand, log id: cefb307
>2018-05-30 05:55:02,675-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [72c6a14e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 72c6a14e, Job ID: 23c1e0ae-113a-45cc-bd05-df5f4099d2f2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 05:55:02,679-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [72c6a14e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 340727ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 05:55:02,682-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [72c6a14e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 340727ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 05:55:02,692-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [31817824] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 05:55:02,693-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [31817824] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7885527] Autorecovering 2 hosts
>2018-05-30 06:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7885527] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:00:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7951846c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:00:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7951846c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:00:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7951846c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:00:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7951846c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:00:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7951846c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45f14d30
>2018-05-30 06:00:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7951846c] FINISH, SetVdsStatusVDSCommand, log id: 45f14d30
>2018-05-30 06:00:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7951846c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:00:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7951846c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:00:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7951846c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:00:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4850022e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:00:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4850022e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:00:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4850022e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:00:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4850022e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:00:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4850022e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 310d8d0e
>2018-05-30 06:00:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4850022e] FINISH, SetVdsStatusVDSCommand, log id: 310d8d0e
>2018-05-30 06:00:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [4850022e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3e356789
>2018-05-30 06:00:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [4850022e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3e356789
>2018-05-30 06:00:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4850022e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:00:00,104-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4850022e] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 4850022e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-30 06:00:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4850022e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:00:00,308-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [620177ba] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 76cc2d6a
>2018-05-30 06:00:00,310-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [620177ba] FINISH, GetHardwareInfoVDSCommand, log id: 76cc2d6a
>2018-05-30 06:00:00,323-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4025e414] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:00:00,328-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [2bab774a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:00:00,333-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [18affc7d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:00:00,339-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [18affc7d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 430919bc
>2018-05-30 06:00:00,711-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [18affc7d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:00:00,711-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [18affc7d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 430919bc
>2018-05-30 06:00:00,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [18affc7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:00:00,717-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [18affc7d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:00:00,721-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [18affc7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:00:00,722-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [18affc7d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:00:00,726-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [18affc7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:00:00,726-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [18affc7d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:00:00,729-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [18affc7d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:00:00,732-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [18affc7d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 18affc7d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:00:00,735-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3907417f
>2018-05-30 06:00:00,735-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:00:00,836-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:00:00,836-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:00:00,836-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:00:00,836-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:00:00,836-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3907417f
>2018-05-30 06:00:00,836-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:00:00,850-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [490886f0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:00:00,851-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [490886f0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ee2866e
>2018-05-30 06:00:00,853-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [490886f0] FINISH, SetVdsStatusVDSCommand, log id: ee2866e
>2018-05-30 06:00:00,882-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [490886f0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 490886f0, Job ID: 40126c1b-05c7-46e1-bf56-240b6e77d0ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:00:00,887-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [490886f0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2bab774a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:00:00,889-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [490886f0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2bab774a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:00:00,899-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [1c4b5051] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:00:00,901-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [1c4b5051] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:00:02,859-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [678ca7ba] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1e57a4e
>2018-05-30 06:00:02,860-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [678ca7ba] FINISH, GetHardwareInfoVDSCommand, log id: 1e57a4e
>2018-05-30 06:00:02,873-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [73e8c13d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:00:02,882-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [313139bb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:00:02,887-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1eb0a492] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:00:02,894-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1eb0a492] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3c14ef7f
>2018-05-30 06:00:03,269-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1eb0a492] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:00:03,269-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1eb0a492] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3c14ef7f
>2018-05-30 06:00:03,273-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1eb0a492] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:00:03,274-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1eb0a492] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:00:03,277-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1eb0a492] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:00:03,278-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1eb0a492] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:00:03,282-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1eb0a492] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:00:03,282-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1eb0a492] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:00:03,285-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1eb0a492] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:00:03,288-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1eb0a492] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1eb0a492, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:00:03,291-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 76e77f6b
>2018-05-30 06:00:03,291-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:00:03,384-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:00:03,384-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:00:03,384-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:00:03,384-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:00:03,384-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 76e77f6b
>2018-05-30 06:00:03,384-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:00:03,397-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [27350676] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:00:03,399-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [27350676] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c15236c
>2018-05-30 06:00:03,401-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [27350676] FINISH, SetVdsStatusVDSCommand, log id: 2c15236c
>2018-05-30 06:00:03,406-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [27350676] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 27350676, Job ID: 471380ac-9503-4df4-8119-3e7d40233fef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:00:03,410-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [27350676] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 313139bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:00:03,412-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [27350676] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 313139bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:00:03,423-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [11525346] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:00:03,424-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [11525346] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6566e070] Autorecovering 2 hosts
>2018-05-30 06:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6566e070] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:05:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3632109c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3632109c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3632109c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:05:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3632109c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:05:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3632109c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bb6e5b5
>2018-05-30 06:05:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3632109c] FINISH, SetVdsStatusVDSCommand, log id: 3bb6e5b5
>2018-05-30 06:05:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3632109c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:05:00,079-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3632109c] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 3632109c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered.
>2018-05-30 06:05:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3632109c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:05:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3632109c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:05:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55427bd3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:05:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55427bd3] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:05:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55427bd3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:05:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55427bd3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:05:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [55427bd3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54e470e3
>2018-05-30 06:05:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [55427bd3] FINISH, SetVdsStatusVDSCommand, log id: 54e470e3
>2018-05-30 06:05:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [55427bd3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4f927111
>2018-05-30 06:05:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [55427bd3] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4f927111
>2018-05-30 06:05:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55427bd3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:05:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55427bd3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:05:01,085-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1b8336f7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 413ecc88
>2018-05-30 06:05:01,087-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1b8336f7] FINISH, GetHardwareInfoVDSCommand, log id: 413ecc88
>2018-05-30 06:05:01,100-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [1b786b87] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:05:01,109-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [729957f9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:05:01,114-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2a5461db] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:05:01,175-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2a5461db] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7031c34a
>2018-05-30 06:05:01,558-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2a5461db] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:05:01,558-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2a5461db] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7031c34a
>2018-05-30 06:05:01,562-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2a5461db] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:05:01,562-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2a5461db] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:05:01,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2a5461db] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:05:01,567-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2a5461db] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:05:01,570-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2a5461db] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:05:01,571-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2a5461db] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:05:01,574-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2a5461db] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:05:01,577-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2a5461db] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2a5461db, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:05:01,583-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3a1640af
>2018-05-30 06:05:01,583-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:05:01,616-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [4964136e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 37393309
>2018-05-30 06:05:01,617-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [4964136e] FINISH, GetHardwareInfoVDSCommand, log id: 37393309
>2018-05-30 06:05:01,630-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7d887db5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:05:01,634-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [53e2783e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:05:01,639-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5a39b582] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:05:01,645-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5a39b582] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 19abaf41
>2018-05-30 06:05:01,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:05:01,682-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:05:01,683-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:05:01,683-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:05:01,683-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3a1640af
>2018-05-30 06:05:01,683-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:05:01,695-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [f0acb1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:05:01,696-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [f0acb1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49d92a18
>2018-05-30 06:05:01,698-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [f0acb1] FINISH, SetVdsStatusVDSCommand, log id: 49d92a18
>2018-05-30 06:05:01,729-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [f0acb1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: f0acb1, Job ID: f13ca29a-c1a4-4d4d-9ba8-cd195a71798f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:05:01,733-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [f0acb1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 729957f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:05:01,735-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [f0acb1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 729957f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:05:01,745-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [532ae93d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:05:01,746-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [532ae93d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:05:02,017-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5a39b582] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:05:02,017-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5a39b582] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 19abaf41
>2018-05-30 06:05:02,021-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5a39b582] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:05:02,021-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5a39b582] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:05:02,025-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5a39b582] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:05:02,026-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5a39b582] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:05:02,030-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5a39b582] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:05:02,031-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5a39b582] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:05:02,034-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5a39b582] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:05:02,036-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5a39b582] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5a39b582, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:05:02,039-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5cef7c3a
>2018-05-30 06:05:02,039-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:05:02,138-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:05:02,138-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:05:02,138-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:05:02,138-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:05:02,138-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5cef7c3a
>2018-05-30 06:05:02,138-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:05:02,151-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [9130d92] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:05:02,152-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [9130d92] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 165b96d7
>2018-05-30 06:05:02,154-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [9130d92] FINISH, SetVdsStatusVDSCommand, log id: 165b96d7
>2018-05-30 06:05:02,160-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [9130d92] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 9130d92, Job ID: d880b4bb-af10-4725-b832-0592dd62b355, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:05:02,164-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [9130d92] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 53e2783e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:05:02,167-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [9130d92] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 53e2783e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:05:02,177-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [253364f2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:05:02,179-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [253364f2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6b9dabdc] Autorecovering 2 hosts
>2018-05-30 06:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6b9dabdc] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:10:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [46d8ca6a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:10:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [46d8ca6a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:10:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [46d8ca6a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:10:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [46d8ca6a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:10:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [46d8ca6a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 71b9d456
>2018-05-30 06:10:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [46d8ca6a] FINISH, SetVdsStatusVDSCommand, log id: 71b9d456
>2018-05-30 06:10:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [46d8ca6a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:10:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [46d8ca6a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:10:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [46d8ca6a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:10:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [77d55822] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:10:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [77d55822] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:10:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [77d55822] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:10:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [77d55822] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:10:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [77d55822] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8b426bf
>2018-05-30 06:10:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [77d55822] FINISH, SetVdsStatusVDSCommand, log id: 8b426bf
>2018-05-30 06:10:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [77d55822] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1643f28
>2018-05-30 06:10:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [77d55822] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1643f28
>2018-05-30 06:10:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [77d55822] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:10:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [77d55822] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:10:02,395-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [253f4490] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2fd5a23c
>2018-05-30 06:10:02,396-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [253f4490] FINISH, GetHardwareInfoVDSCommand, log id: 2fd5a23c
>2018-05-30 06:10:02,410-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [1c2bbe77] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:10:02,419-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [36d039cd] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:10:02,425-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2bdc56a5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:10:02,437-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2bdc56a5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 448c6e15
>2018-05-30 06:10:02,819-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2bdc56a5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:10:02,819-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2bdc56a5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 448c6e15
>2018-05-30 06:10:02,821-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5ac40a85] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5015ee6f
>2018-05-30 06:10:02,823-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5ac40a85] FINISH, GetHardwareInfoVDSCommand, log id: 5015ee6f
>2018-05-30 06:10:02,825-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2bdc56a5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:10:02,826-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2bdc56a5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:10:02,830-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2bdc56a5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:10:02,830-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2bdc56a5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:10:02,834-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2bdc56a5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:10:02,835-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2bdc56a5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:10:02,837-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [65e58eb3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:10:02,839-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2bdc56a5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:10:02,842-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [41519d0c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:10:02,842-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2bdc56a5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2bdc56a5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:10:02,847-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [a0a8051] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:10:02,849-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 14737746
>2018-05-30 06:10:02,849-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:10:02,854-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [a0a8051] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3113168b
>2018-05-30 06:10:02,948-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:10:02,948-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:10:02,948-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:10:02,948-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:10:02,948-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 14737746
>2018-05-30 06:10:02,948-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:10:02,961-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [1321b2ba] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:10:02,962-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1321b2ba] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 426e14cb
>2018-05-30 06:10:02,964-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1321b2ba] FINISH, SetVdsStatusVDSCommand, log id: 426e14cb
>2018-05-30 06:10:02,969-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1321b2ba] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1321b2ba, Job ID: 274cd64b-b64c-4125-b53d-30fd4087a285, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:10:02,974-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1321b2ba] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 36d039cd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:10:02,976-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1321b2ba] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 36d039cd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:10:02,986-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [721a73b5] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:10:02,988-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [721a73b5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:10:03,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [a0a8051] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:10:03,226-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [a0a8051] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3113168b
>2018-05-30 06:10:03,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [a0a8051] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:10:03,231-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [a0a8051] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:10:03,234-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [a0a8051] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:10:03,235-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [a0a8051] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:10:03,239-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [a0a8051] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:10:03,240-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [a0a8051] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:10:03,243-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [a0a8051] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:10:03,246-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [a0a8051] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: a0a8051, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:10:03,249-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3d25e958
>2018-05-30 06:10:03,249-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:10:03,349-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:10:03,349-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:10:03,349-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:10:03,349-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:10:03,349-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3d25e958
>2018-05-30 06:10:03,349-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:10:03,363-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [71bd9338] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:10:03,364-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [71bd9338] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 10286dc0
>2018-05-30 06:10:03,367-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [71bd9338] FINISH, SetVdsStatusVDSCommand, log id: 10286dc0
>2018-05-30 06:10:03,372-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [71bd9338] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 71bd9338, Job ID: e0d41082-5cc7-4cbb-9172-bddfccbb9dbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:10:03,377-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [71bd9338] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 41519d0c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:10:03,379-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [71bd9338] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 41519d0c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:10:03,390-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [1aee8e99] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:10:03,391-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [1aee8e99] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [28b5d657] Autorecovering 2 hosts
>2018-05-30 06:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [28b5d657] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:15:00,038-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [68ad9992] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:15:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [68ad9992] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:15:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [68ad9992] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:15:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [68ad9992] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:15:00,046-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [68ad9992] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1e8f1044
>2018-05-30 06:15:00,049-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [68ad9992] FINISH, SetVdsStatusVDSCommand, log id: 1e8f1044
>2018-05-30 06:15:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [68ad9992] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:15:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [68ad9992] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:15:00,052-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [68ad9992] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:15:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6b2ba46d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:15:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6b2ba46d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:15:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6b2ba46d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:15:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6b2ba46d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:15:00,065-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6b2ba46d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 632393f7
>2018-05-30 06:15:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6b2ba46d] FINISH, SetVdsStatusVDSCommand, log id: 632393f7
>2018-05-30 06:15:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [6b2ba46d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 692a1cae
>2018-05-30 06:15:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [6b2ba46d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 692a1cae
>2018-05-30 06:15:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6b2ba46d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:15:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6b2ba46d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:15:00,641-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [38344d22] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1321586f
>2018-05-30 06:15:00,643-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [38344d22] FINISH, GetHardwareInfoVDSCommand, log id: 1321586f
>2018-05-30 06:15:00,655-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [10b3db0d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:15:00,664-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [17bb5d12] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:15:00,669-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [29374b39] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:15:00,680-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [29374b39] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 125f2259
>2018-05-30 06:15:01,010-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2c7dc724] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2bc3039e
>2018-05-30 06:15:01,012-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2c7dc724] FINISH, GetHardwareInfoVDSCommand, log id: 2bc3039e
>2018-05-30 06:15:01,024-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [5007ca70] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:15:01,032-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5688878f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:15:01,038-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [160e82c1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:15:01,050-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [160e82c1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7d12b098
>2018-05-30 06:15:01,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [29374b39] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:15:01,061-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [29374b39] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 125f2259
>2018-05-30 06:15:01,065-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [29374b39] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:15:01,065-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [29374b39] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:15:01,069-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [29374b39] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:15:01,070-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [29374b39] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:15:01,074-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [29374b39] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:15:01,074-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [29374b39] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:15:01,078-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [29374b39] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:15:01,080-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [29374b39] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 29374b39, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:15:01,087-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1131f2b3
>2018-05-30 06:15:01,087-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:15:01,186-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:15:01,186-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:15:01,186-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:15:01,187-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:15:01,187-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1131f2b3
>2018-05-30 06:15:01,187-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:15:01,199-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [64c85782] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:15:01,200-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [64c85782] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d4757e7
>2018-05-30 06:15:01,203-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [64c85782] FINISH, SetVdsStatusVDSCommand, log id: d4757e7
>2018-05-30 06:15:01,211-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [64c85782] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 64c85782, Job ID: cbbd9137-a02d-4a1d-b9ad-2a13a663d373, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:15:01,216-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [64c85782] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17bb5d12, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:15:01,218-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [64c85782] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17bb5d12, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:15:01,229-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [36753e76] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:15:01,230-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [36753e76] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:15:02,413-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [160e82c1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:15:02,413-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [160e82c1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7d12b098
>2018-05-30 06:15:02,417-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [160e82c1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:15:02,417-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [160e82c1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:15:02,421-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [160e82c1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:15:02,422-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [160e82c1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:15:02,425-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [160e82c1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:15:02,426-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [160e82c1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:15:02,429-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [160e82c1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:15:02,432-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [160e82c1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 160e82c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:15:02,434-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2b9eaa4a
>2018-05-30 06:15:02,434-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:15:02,530-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:15:02,530-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:15:02,530-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:15:02,530-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:15:02,530-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2b9eaa4a
>2018-05-30 06:15:02,530-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:15:02,543-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [226fd8d3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:15:02,544-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [226fd8d3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4067d2d1
>2018-05-30 06:15:02,546-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [226fd8d3] FINISH, SetVdsStatusVDSCommand, log id: 4067d2d1
>2018-05-30 06:15:02,551-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [226fd8d3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 226fd8d3, Job ID: 426c8d4b-5224-4001-8a37-f2232d1e9f82, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:15:02,556-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [226fd8d3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5688878f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:15:02,558-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [226fd8d3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5688878f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:15:02,569-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [680d3373] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:15:02,570-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [680d3373] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [396b605] Autorecovering 2 hosts
>2018-05-30 06:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [396b605] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:20:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49f7e024] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:20:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49f7e024] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:20:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49f7e024] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:20:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49f7e024] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:20:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [49f7e024] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3985150d
>2018-05-30 06:20:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [49f7e024] FINISH, SetVdsStatusVDSCommand, log id: 3985150d
>2018-05-30 06:20:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49f7e024] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:20:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49f7e024] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:20:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [49f7e024] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:20:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [548986fb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:20:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [548986fb] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:20:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [548986fb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:20:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [548986fb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:20:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [548986fb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7d0d1ab0
>2018-05-30 06:20:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [548986fb] FINISH, SetVdsStatusVDSCommand, log id: 7d0d1ab0
>2018-05-30 06:20:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [548986fb] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2a392008
>2018-05-30 06:20:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [548986fb] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2a392008
>2018-05-30 06:20:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [548986fb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:20:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [548986fb] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:20:00,263-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4e52567f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 281c9a2b
>2018-05-30 06:20:00,265-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4e52567f] FINISH, GetHardwareInfoVDSCommand, log id: 281c9a2b
>2018-05-30 06:20:00,278-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [662de495] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:20:00,286-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [45eb72b1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:20:00,292-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [63e1722c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:20:00,304-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [63e1722c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2dfa0b06
>2018-05-30 06:20:00,681-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [63e1722c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:20:00,681-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [63e1722c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2dfa0b06
>2018-05-30 06:20:00,685-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [63e1722c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:20:00,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [63e1722c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:20:00,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [63e1722c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:20:00,690-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [63e1722c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:20:00,694-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [63e1722c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:20:00,695-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [63e1722c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:20:00,697-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [63e1722c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:20:00,699-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [63e1722c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 63e1722c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:20:00,706-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1377c37d
>2018-05-30 06:20:00,706-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:20:00,803-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:20:00,803-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:20:00,803-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:20:00,804-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:20:00,804-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1377c37d
>2018-05-30 06:20:00,804-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:20:00,817-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [62bbf2ef] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:20:00,818-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [62bbf2ef] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: c0d39aa
>2018-05-30 06:20:00,820-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [62bbf2ef] FINISH, SetVdsStatusVDSCommand, log id: c0d39aa
>2018-05-30 06:20:00,854-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [62bbf2ef] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 62bbf2ef, Job ID: 830a9115-756b-40c0-8806-9d05bfff0667, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:20:00,859-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [62bbf2ef] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 45eb72b1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:20:00,861-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [62bbf2ef] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 45eb72b1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:20:00,871-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [2569640e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:20:00,872-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [2569640e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:20:01,881-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3bd2883] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 835968e
>2018-05-30 06:20:01,883-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3bd2883] FINISH, GetHardwareInfoVDSCommand, log id: 835968e
>2018-05-30 06:20:01,896-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [91acef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:20:01,904-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [404c93c2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:20:01,910-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6269e8c8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:20:01,921-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6269e8c8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6e3be37b
>2018-05-30 06:20:02,302-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6269e8c8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:20:02,302-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6269e8c8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6e3be37b
>2018-05-30 06:20:02,305-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6269e8c8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:20:02,306-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6269e8c8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:20:02,310-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6269e8c8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:20:02,311-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6269e8c8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:20:02,314-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6269e8c8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:20:02,315-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6269e8c8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:20:02,318-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6269e8c8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:20:02,321-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6269e8c8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6269e8c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:20:02,328-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 21e972a1
>2018-05-30 06:20:02,328-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:20:02,425-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:20:02,425-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:20:02,425-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:20:02,425-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:20:02,425-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 21e972a1
>2018-05-30 06:20:02,425-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:20:02,439-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [2ab933a3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:20:02,440-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2ab933a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 51014f7
>2018-05-30 06:20:02,443-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2ab933a3] FINISH, SetVdsStatusVDSCommand, log id: 51014f7
>2018-05-30 06:20:02,476-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ab933a3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2ab933a3, Job ID: cc7615ed-e7a8-4c50-b478-3551c0ab553f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:20:02,481-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ab933a3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 404c93c2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:20:02,483-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ab933a3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 404c93c2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:20:02,494-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [5ef6df05] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:20:02,495-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [5ef6df05] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:25:00,007-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [593fd8b1] Autorecovering 2 hosts
>2018-05-30 06:25:00,007-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [593fd8b1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:25:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5ed34a17] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:25:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5ed34a17] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:25:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5ed34a17] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:25:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5ed34a17] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:25:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5ed34a17] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e4336e4
>2018-05-30 06:25:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5ed34a17] FINISH, SetVdsStatusVDSCommand, log id: 5e4336e4
>2018-05-30 06:25:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5ed34a17] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:25:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5ed34a17] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:25:00,082-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5ed34a17] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:25:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [74fe8376] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:25:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [74fe8376] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:25:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [74fe8376] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:25:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [74fe8376] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:25:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [74fe8376] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 58420254
>2018-05-30 06:25:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [74fe8376] FINISH, SetVdsStatusVDSCommand, log id: 58420254
>2018-05-30 06:25:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [74fe8376] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2dbe2d7e
>2018-05-30 06:25:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [74fe8376] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2dbe2d7e
>2018-05-30 06:25:00,102-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [74fe8376] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:25:00,104-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [74fe8376] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:25:01,564-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1f1d0ebe] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2da38441
>2018-05-30 06:25:01,566-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1f1d0ebe] FINISH, GetHardwareInfoVDSCommand, log id: 2da38441
>2018-05-30 06:25:01,579-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [43776a5e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:25:01,588-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [7b162c29] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:25:01,593-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7661f50c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:25:01,604-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7661f50c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 396fc42b
>2018-05-30 06:25:01,977-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7661f50c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:25:01,977-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7661f50c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 396fc42b
>2018-05-30 06:25:01,981-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7661f50c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:25:01,982-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7661f50c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:25:01,986-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7661f50c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:25:01,986-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7661f50c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:25:01,991-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7661f50c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:25:01,991-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7661f50c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:25:01,994-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7661f50c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:25:01,998-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7661f50c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7661f50c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:25:02,005-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 54f9b9da
>2018-05-30 06:25:02,005-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:25:02,100-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:25:02,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:25:02,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:25:02,100-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:25:02,100-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 54f9b9da
>2018-05-30 06:25:02,100-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:25:02,114-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [17434061] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:25:02,115-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [17434061] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18418b0
>2018-05-30 06:25:02,117-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [17434061] FINISH, SetVdsStatusVDSCommand, log id: 18418b0
>2018-05-30 06:25:02,150-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [17434061] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 17434061, Job ID: a8e9920d-1ddb-4313-a9f8-6b019d691175, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:25:02,154-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [17434061] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7b162c29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:25:02,157-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [17434061] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7b162c29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:25:02,168-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [34a4f041] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:25:02,169-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [34a4f041] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:25:03,912-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [41761b57] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1f2338bc
>2018-05-30 06:25:03,913-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [41761b57] FINISH, GetHardwareInfoVDSCommand, log id: 1f2338bc
>2018-05-30 06:25:03,927-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6723ad5c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:25:03,932-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [12eddb9f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:25:03,937-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [36a1777c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:25:03,944-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [36a1777c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 90dc6d4
>2018-05-30 06:25:04,319-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36a1777c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:25:04,319-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [36a1777c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 90dc6d4
>2018-05-30 06:25:04,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36a1777c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:25:04,324-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [36a1777c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:25:04,328-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36a1777c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:25:04,328-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [36a1777c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:25:04,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36a1777c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:25:04,333-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [36a1777c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:25:04,336-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [36a1777c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:25:04,338-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36a1777c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 36a1777c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:25:04,342-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3e32ce28
>2018-05-30 06:25:04,342-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:25:04,439-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:25:04,439-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:25:04,439-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:25:04,439-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:25:04,439-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3e32ce28
>2018-05-30 06:25:04,439-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:25:04,453-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [2190d920] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:25:04,454-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2190d920] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f69dfe7
>2018-05-30 06:25:04,456-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2190d920] FINISH, SetVdsStatusVDSCommand, log id: 3f69dfe7
>2018-05-30 06:25:04,462-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2190d920] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2190d920, Job ID: fb63bbbb-7ba5-40f1-960a-656ce4b434a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:25:04,466-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2190d920] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 12eddb9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:25:04,469-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2190d920] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 12eddb9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:25:04,479-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [4bd21ce4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:25:04,480-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [4bd21ce4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6daca0f2] Autorecovering 2 hosts
>2018-05-30 06:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6daca0f2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:30:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [22512415] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:30:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [22512415] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:30:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [22512415] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:30:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [22512415] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:30:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [22512415] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f935a73
>2018-05-30 06:30:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [22512415] FINISH, SetVdsStatusVDSCommand, log id: 7f935a73
>2018-05-30 06:30:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [22512415] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:30:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [22512415] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:30:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [22512415] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:30:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2c6defec] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:30:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2c6defec] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:30:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2c6defec] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:30:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2c6defec] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:30:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2c6defec] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 729ac2cd
>2018-05-30 06:30:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2c6defec] FINISH, SetVdsStatusVDSCommand, log id: 729ac2cd
>2018-05-30 06:30:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [2c6defec] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6399f10d
>2018-05-30 06:30:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [2c6defec] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6399f10d
>2018-05-30 06:30:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2c6defec] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:30:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2c6defec] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:30:02,148-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2d3fe737] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 435fab1f
>2018-05-30 06:30:02,150-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2d3fe737] FINISH, GetHardwareInfoVDSCommand, log id: 435fab1f
>2018-05-30 06:30:02,163-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [5261362c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:30:02,171-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4b8d2cdb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:30:02,176-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7cf4ad9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:30:02,188-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7cf4ad9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 41cb6128
>2018-05-30 06:30:02,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7cf4ad9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:30:02,567-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7cf4ad9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 41cb6128
>2018-05-30 06:30:02,571-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7cf4ad9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:30:02,571-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7cf4ad9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:30:02,576-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7cf4ad9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:30:02,576-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7cf4ad9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:30:02,580-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7cf4ad9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:30:02,581-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7cf4ad9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:30:02,584-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7cf4ad9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:30:02,586-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7cf4ad9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7cf4ad9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:30:02,593-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4aff4f01
>2018-05-30 06:30:02,593-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:30:02,689-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:30:02,689-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:30:02,689-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:30:02,689-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:30:02,689-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4aff4f01
>2018-05-30 06:30:02,689-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:30:02,702-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [275afff8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:30:02,703-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [275afff8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 349a9988
>2018-05-30 06:30:02,706-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [275afff8] FINISH, SetVdsStatusVDSCommand, log id: 349a9988
>2018-05-30 06:30:02,741-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [275afff8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 275afff8, Job ID: 49be110f-a34a-4ea9-9f99-df4e8408a446, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:30:02,746-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [275afff8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4b8d2cdb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:30:02,748-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [275afff8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4b8d2cdb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:30:02,759-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [4ed388b9] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:30:02,760-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [4ed388b9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:30:02,867-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [62818476] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 64430148
>2018-05-30 06:30:02,869-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [62818476] FINISH, GetHardwareInfoVDSCommand, log id: 64430148
>2018-05-30 06:30:02,882-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [1a922ce3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:30:02,890-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [83d24e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:30:02,895-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [66e8dbfe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:30:02,906-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [66e8dbfe] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3ee6f520
>2018-05-30 06:30:03,276-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [66e8dbfe] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:30:03,276-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [66e8dbfe] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3ee6f520
>2018-05-30 06:30:03,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [66e8dbfe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:30:03,280-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [66e8dbfe] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:30:03,284-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [66e8dbfe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:30:03,285-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [66e8dbfe] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:30:03,289-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [66e8dbfe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:30:03,289-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [66e8dbfe] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:30:03,293-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [66e8dbfe] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:30:03,295-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [66e8dbfe] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 66e8dbfe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:30:03,302-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2033f1c9
>2018-05-30 06:30:03,302-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:30:03,400-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:30:03,400-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:30:03,400-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:30:03,400-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:30:03,400-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2033f1c9
>2018-05-30 06:30:03,400-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:30:03,413-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [31ef0b58] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:30:03,415-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [31ef0b58] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49743a41
>2018-05-30 06:30:03,417-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [31ef0b58] FINISH, SetVdsStatusVDSCommand, log id: 49743a41
>2018-05-30 06:30:03,445-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [31ef0b58] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 31ef0b58, Job ID: eceaa3bd-adaa-47a9-814e-f7bc1a825777, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:30:03,450-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [31ef0b58] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 83d24e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:30:03,452-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [31ef0b58] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 83d24e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:30:03,463-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [146f121e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:30:03,464-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [146f121e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2a62c325] Autorecovering 2 hosts
>2018-05-30 06:35:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2a62c325] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:35:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34b28141] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:35:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34b28141] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:35:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34b28141] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:35:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34b28141] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:35:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [34b28141] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4892c060
>2018-05-30 06:35:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [34b28141] FINISH, SetVdsStatusVDSCommand, log id: 4892c060
>2018-05-30 06:35:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34b28141] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:35:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34b28141] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:35:00,073-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [34b28141] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:35:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c9b959e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:35:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c9b959e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:35:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c9b959e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:35:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c9b959e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:35:00,086-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5c9b959e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4f97ce9c
>2018-05-30 06:35:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5c9b959e] FINISH, SetVdsStatusVDSCommand, log id: 4f97ce9c
>2018-05-30 06:35:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [5c9b959e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5e152abf
>2018-05-30 06:35:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [5c9b959e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5e152abf
>2018-05-30 06:35:00,092-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c9b959e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:35:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c9b959e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:35:00,462-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3068b85e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2df25dc5
>2018-05-30 06:35:00,464-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3068b85e] FINISH, GetHardwareInfoVDSCommand, log id: 2df25dc5
>2018-05-30 06:35:00,477-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [5c6b7e4b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:35:00,485-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [6726d23] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:35:00,490-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [7ce8a536] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:35:00,496-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [7ce8a536] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5432c6c6
>2018-05-30 06:35:00,880-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7ce8a536] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:35:00,880-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [7ce8a536] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5432c6c6
>2018-05-30 06:35:00,884-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7ce8a536] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:35:00,885-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7ce8a536] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:35:00,888-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7ce8a536] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:35:00,889-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7ce8a536] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:35:00,892-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7ce8a536] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:35:00,893-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7ce8a536] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:35:00,896-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [7ce8a536] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:35:00,898-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7ce8a536] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7ce8a536, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:35:00,901-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2ca22ca4
>2018-05-30 06:35:00,901-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:35:00,998-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:35:00,998-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:35:00,998-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:35:00,998-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:35:00,998-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2ca22ca4
>2018-05-30 06:35:00,998-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:35:01,011-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [592a3921] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:35:01,012-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [592a3921] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f0a6546
>2018-05-30 06:35:01,014-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [592a3921] FINISH, SetVdsStatusVDSCommand, log id: 7f0a6546
>2018-05-30 06:35:01,044-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [592a3921] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 592a3921, Job ID: ff6b97ac-473a-4861-a3e7-1bf8602fa8f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:35:01,049-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [592a3921] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6726d23, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:35:01,051-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [592a3921] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6726d23, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:35:01,062-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7177f2f0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:35:01,063-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7177f2f0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:35:01,182-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [79aec68b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 22abe3fc
>2018-05-30 06:35:01,184-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [79aec68b] FINISH, GetHardwareInfoVDSCommand, log id: 22abe3fc
>2018-05-30 06:35:01,196-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6e7061bc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:35:01,201-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [c82c31f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:35:01,205-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [41ec06b0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:35:01,212-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [41ec06b0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 78fd3fc8
>2018-05-30 06:35:03,532-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [41ec06b0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:35:03,532-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [41ec06b0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 78fd3fc8
>2018-05-30 06:35:03,536-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [41ec06b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:35:03,537-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [41ec06b0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:35:03,541-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [41ec06b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:35:03,541-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [41ec06b0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:35:03,545-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [41ec06b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:35:03,546-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [41ec06b0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:35:03,549-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [41ec06b0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:35:03,552-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [41ec06b0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 41ec06b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:35:03,555-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7940caf9
>2018-05-30 06:35:03,555-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:35:03,651-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:35:03,651-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:35:03,651-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:35:03,651-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:35:03,651-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7940caf9
>2018-05-30 06:35:03,651-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:35:03,664-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [65a9f388] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:35:03,666-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [65a9f388] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5ad004db
>2018-05-30 06:35:03,668-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [65a9f388] FINISH, SetVdsStatusVDSCommand, log id: 5ad004db
>2018-05-30 06:35:03,700-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [65a9f388] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 65a9f388, Job ID: 04444bf0-ea35-44bb-ac59-5df6e0e1a628, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:35:03,705-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [65a9f388] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: c82c31f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:35:03,707-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [65a9f388] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: c82c31f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:35:03,717-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [218cb8a0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:35:03,718-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [218cb8a0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [77aa42a4] Autorecovering 2 hosts
>2018-05-30 06:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [77aa42a4] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:40:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f16f4a1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f16f4a1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f16f4a1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f16f4a1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:40:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6f16f4a1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 79ec384e
>2018-05-30 06:40:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6f16f4a1] FINISH, SetVdsStatusVDSCommand, log id: 79ec384e
>2018-05-30 06:40:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f16f4a1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:40:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f16f4a1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:40:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6f16f4a1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:40:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [237c18b1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [237c18b1] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [237c18b1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:40:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [237c18b1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:40:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [237c18b1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 40d2910d
>2018-05-30 06:40:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [237c18b1] FINISH, SetVdsStatusVDSCommand, log id: 40d2910d
>2018-05-30 06:40:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [237c18b1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1d1813b2
>2018-05-30 06:40:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [237c18b1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1d1813b2
>2018-05-30 06:40:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [237c18b1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:40:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [237c18b1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:40:01,518-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [27fb8b71] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 546a4aa1
>2018-05-30 06:40:01,520-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [27fb8b71] FINISH, GetHardwareInfoVDSCommand, log id: 546a4aa1
>2018-05-30 06:40:01,532-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [7d716505] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:40:01,537-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [72e80d3e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:40:01,542-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4c9b5ae0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:40:01,603-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4c9b5ae0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 70aae753
>2018-05-30 06:40:01,720-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1ca5071c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6650d88b
>2018-05-30 06:40:01,722-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1ca5071c] FINISH, GetHardwareInfoVDSCommand, log id: 6650d88b
>2018-05-30 06:40:01,737-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [705a5750] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:40:01,741-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5c4562f3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:40:01,746-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7f51e594] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:40:01,753-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7f51e594] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3f11f2eb
>2018-05-30 06:40:01,974-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c9b5ae0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:40:01,974-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4c9b5ae0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 70aae753
>2018-05-30 06:40:01,978-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c9b5ae0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:40:01,979-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4c9b5ae0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:40:01,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c9b5ae0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:40:01,983-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4c9b5ae0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:40:01,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c9b5ae0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:40:01,988-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4c9b5ae0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:40:01,991-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4c9b5ae0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:40:01,994-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c9b5ae0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4c9b5ae0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:40:02,002-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 265de894
>2018-05-30 06:40:02,002-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:40:02,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:40:02,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:40:02,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:40:02,099-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:40:02,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 265de894
>2018-05-30 06:40:02,099-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:40:02,112-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [62e472a8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:40:02,113-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [62e472a8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53ab147
>2018-05-30 06:40:02,115-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [62e472a8] FINISH, SetVdsStatusVDSCommand, log id: 53ab147
>2018-05-30 06:40:02,124-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [62e472a8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 62e472a8, Job ID: e93c3d6b-441e-4f59-ba12-084de009ec2d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:40:02,125-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7f51e594] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:40:02,125-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7f51e594] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3f11f2eb
>2018-05-30 06:40:02,129-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [62e472a8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 72e80d3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:40:02,130-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7f51e594] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:40:02,130-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7f51e594] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:40:02,131-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [62e472a8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 72e80d3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:40:02,135-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7f51e594] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:40:02,135-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7f51e594] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:40:02,139-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7f51e594] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:40:02,140-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7f51e594] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:40:02,143-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [137131c3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:40:02,144-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [137131c3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:40:02,144-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7f51e594] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:40:02,147-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7f51e594] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7f51e594, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:40:02,150-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 73d6ce3c
>2018-05-30 06:40:02,150-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:40:02,246-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:40:02,246-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:40:02,246-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:40:02,246-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:40:02,246-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 73d6ce3c
>2018-05-30 06:40:02,246-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:40:02,260-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [580961be] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:40:02,262-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [580961be] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2db780c3
>2018-05-30 06:40:02,264-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [580961be] FINISH, SetVdsStatusVDSCommand, log id: 2db780c3
>2018-05-30 06:40:02,273-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [580961be] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 580961be, Job ID: 384ca8f7-6b62-4601-9698-0eed676e01eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:40:02,277-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [580961be] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5c4562f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:40:02,279-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [580961be] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5c4562f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:40:02,290-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [39917e5b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:40:02,291-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [39917e5b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5cbc1b40] Autorecovering 2 hosts
>2018-05-30 06:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5cbc1b40] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:45:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3bfb13ea] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3bfb13ea] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3bfb13ea] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:45:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3bfb13ea] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:45:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3bfb13ea] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bdb62e0
>2018-05-30 06:45:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3bfb13ea] FINISH, SetVdsStatusVDSCommand, log id: 3bdb62e0
>2018-05-30 06:45:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3bfb13ea] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:45:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3bfb13ea] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:45:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3bfb13ea] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:45:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [69d49c82] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:45:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [69d49c82] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:45:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [69d49c82] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:45:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [69d49c82] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:45:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [69d49c82] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f492315
>2018-05-30 06:45:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [69d49c82] FINISH, SetVdsStatusVDSCommand, log id: 5f492315
>2018-05-30 06:45:00,117-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [69d49c82] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 28e09b1b
>2018-05-30 06:45:00,120-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [69d49c82] FINISH, SetHaMaintenanceModeVDSCommand, log id: 28e09b1b
>2018-05-30 06:45:00,120-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [69d49c82] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:45:00,121-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [69d49c82] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:45:02,864-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5210d619] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: db3b505
>2018-05-30 06:45:02,866-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5210d619] FINISH, GetHardwareInfoVDSCommand, log id: db3b505
>2018-05-30 06:45:02,874-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5bb2cf88] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 700b5aee
>2018-05-30 06:45:02,876-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5bb2cf88] FINISH, GetHardwareInfoVDSCommand, log id: 700b5aee
>2018-05-30 06:45:02,880-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [5e2d4139] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:45:02,889-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [5aac4b94] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:45:02,889-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [ba1a2e4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:45:02,894-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [75e6e032] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:45:02,898-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [52060b2b] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:45:02,903-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [69791422] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:45:02,907-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [75e6e032] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 500fbc86
>2018-05-30 06:45:02,915-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [69791422] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1b91d745
>2018-05-30 06:45:03,287-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [75e6e032] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:45:03,287-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [75e6e032] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 500fbc86
>2018-05-30 06:45:03,291-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [75e6e032] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:45:03,292-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [75e6e032] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:45:03,295-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [75e6e032] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:45:03,296-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [75e6e032] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:45:03,300-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [75e6e032] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:45:03,301-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [75e6e032] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:45:03,304-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [75e6e032] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:45:03,306-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [75e6e032] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 75e6e032, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:45:03,313-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 12ddb0d8
>2018-05-30 06:45:03,313-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:45:03,407-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:45:03,407-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:45:03,407-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:45:03,407-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:45:03,407-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 12ddb0d8
>2018-05-30 06:45:03,407-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:45:03,420-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [48ba65bf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:45:03,421-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [48ba65bf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3388052c
>2018-05-30 06:45:03,424-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [48ba65bf] FINISH, SetVdsStatusVDSCommand, log id: 3388052c
>2018-05-30 06:45:03,456-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [48ba65bf] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 48ba65bf, Job ID: 583e8170-6bd1-4406-b851-d5f780687ce7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:45:03,461-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [48ba65bf] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5aac4b94, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:45:03,464-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [48ba65bf] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5aac4b94, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:45:03,474-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [54aa251f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:45:03,475-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [54aa251f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:45:04,319-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69791422] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:45:04,319-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [69791422] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1b91d745
>2018-05-30 06:45:04,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69791422] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:45:04,323-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [69791422] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:45:04,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69791422] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:45:04,328-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [69791422] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:45:04,331-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69791422] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:45:04,332-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [69791422] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:45:04,335-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [69791422] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:45:04,337-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69791422] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 69791422, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:45:04,340-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 57b82508
>2018-05-30 06:45:04,340-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:45:04,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:45:04,436-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:45:04,436-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:45:04,436-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:45:04,436-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 57b82508
>2018-05-30 06:45:04,436-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:45:04,448-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [7d6c8e48] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:45:04,449-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7d6c8e48] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2a977925
>2018-05-30 06:45:04,451-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7d6c8e48] FINISH, SetVdsStatusVDSCommand, log id: 2a977925
>2018-05-30 06:45:04,457-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7d6c8e48] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7d6c8e48, Job ID: da0c7860-5de0-41ad-a2b8-6a2e2bd67e10, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:45:04,461-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7d6c8e48] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 52060b2b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:45:04,463-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7d6c8e48] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 52060b2b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:45:04,473-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [342394db] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:45:04,474-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [342394db] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5f3029bf] Autorecovering 2 hosts
>2018-05-30 06:50:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5f3029bf] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:50:00,060-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [45415bae] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:50:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [45415bae] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:50:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [45415bae] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:50:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [45415bae] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:50:00,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [45415bae] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31d29741
>2018-05-30 06:50:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [45415bae] FINISH, SetVdsStatusVDSCommand, log id: 31d29741
>2018-05-30 06:50:00,073-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [45415bae] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:50:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [45415bae] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:50:00,074-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [45415bae] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:50:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4947fb95] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:50:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4947fb95] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:50:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4947fb95] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:50:00,086-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4947fb95] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:50:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4947fb95] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 508a50b6
>2018-05-30 06:50:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4947fb95] FINISH, SetVdsStatusVDSCommand, log id: 508a50b6
>2018-05-30 06:50:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4947fb95] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 36830921
>2018-05-30 06:50:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4947fb95] FINISH, SetHaMaintenanceModeVDSCommand, log id: 36830921
>2018-05-30 06:50:00,094-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4947fb95] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:50:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4947fb95] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:50:01,064-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3943ad76] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 619942e0
>2018-05-30 06:50:01,066-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3943ad76] FINISH, GetHardwareInfoVDSCommand, log id: 619942e0
>2018-05-30 06:50:01,079-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [193b281c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:50:01,083-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [62c9272] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:50:01,089-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3b6cbe0d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:50:01,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3b6cbe0d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5b0d23ad
>2018-05-30 06:50:01,470-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3b6cbe0d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:50:01,470-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3b6cbe0d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5b0d23ad
>2018-05-30 06:50:01,474-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3b6cbe0d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:50:01,474-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3b6cbe0d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:50:01,478-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3b6cbe0d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:50:01,479-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3b6cbe0d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:50:01,482-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3b6cbe0d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:50:01,483-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3b6cbe0d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:50:01,487-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3b6cbe0d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:50:01,490-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3b6cbe0d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3b6cbe0d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:50:01,492-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 342aba40
>2018-05-30 06:50:01,492-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:50:01,592-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:50:01,592-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:50:01,592-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:50:01,592-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:50:01,592-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: 342aba40
>2018-05-30 06:50:01,592-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:50:01,604-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5fde0f2e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:50:01,606-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5fde0f2e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31bf53ad
>2018-05-30 06:50:01,608-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5fde0f2e] FINISH, SetVdsStatusVDSCommand, log id: 31bf53ad
>2018-05-30 06:50:01,614-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5fde0f2e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5fde0f2e, Job ID: d633be31-02c3-4a4a-9f60-d151c4b1b7c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:50:01,618-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5fde0f2e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 62c9272, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:50:01,620-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5fde0f2e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 62c9272, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:50:01,631-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [476e2797] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:50:01,632-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [476e2797] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:50:02,081-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5efe00e3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7e9d54da
>2018-05-30 06:50:02,083-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5efe00e3] FINISH, GetHardwareInfoVDSCommand, log id: 7e9d54da
>2018-05-30 06:50:02,097-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [34231990] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:50:02,101-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [2d537acc] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:50:02,107-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2558c7e7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:50:02,114-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2558c7e7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 26462a90
>2018-05-30 06:50:02,483-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2558c7e7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:50:02,483-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2558c7e7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 26462a90
>2018-05-30 06:50:02,487-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2558c7e7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:50:02,488-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2558c7e7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:50:02,491-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2558c7e7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:50:02,492-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2558c7e7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:50:02,496-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2558c7e7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:50:02,497-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2558c7e7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:50:02,499-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2558c7e7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:50:02,502-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2558c7e7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2558c7e7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:50:02,505-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 45ef468e
>2018-05-30 06:50:02,505-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:50:02,604-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:50:02,604-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:50:02,604-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:50:02,604-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:50:02,604-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 45ef468e
>2018-05-30 06:50:02,604-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:50:02,617-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [71cdc935] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:50:02,618-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [71cdc935] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3967b4eb
>2018-05-30 06:50:02,620-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [71cdc935] FINISH, SetVdsStatusVDSCommand, log id: 3967b4eb
>2018-05-30 06:50:02,626-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [71cdc935] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 71cdc935, Job ID: ba439923-8c90-4f2a-a06c-b10f1f1fc6cf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:50:02,630-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [71cdc935] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2d537acc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:50:02,632-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [71cdc935] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2d537acc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:50:02,643-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [4257660a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:50:02,644-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [4257660a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:54:43,570-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler2) [14aa7392] Attempting to update VMs/Templates Ovf.
>2018-05-30 06:54:43,572-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 06:54:43,572-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 06:54:43,573-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:54:43,577-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-30 06:54:43,580-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-30 06:54:43,580-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Attempting to update template OVFs in Data Center 'Default'
>2018-05-30 06:54:43,580-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-30 06:54:43,580-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 06:54:43,581-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 06:54:43,581-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 06:55:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7e83ac54] Autorecovering 2 hosts
>2018-05-30 06:55:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7e83ac54] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:55:00,030-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3425631b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:55:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3425631b] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:55:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3425631b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:55:00,037-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3425631b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:55:00,038-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3425631b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7819d24f
>2018-05-30 06:55:00,041-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3425631b] FINISH, SetVdsStatusVDSCommand, log id: 7819d24f
>2018-05-30 06:55:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3425631b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:55:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3425631b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 06:55:00,043-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [3425631b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:55:00,049-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c2da757] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:55:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c2da757] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 06:55:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c2da757] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:55:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c2da757] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:55:00,057-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5c2da757] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 51c4e11f
>2018-05-30 06:55:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5c2da757] FINISH, SetVdsStatusVDSCommand, log id: 51c4e11f
>2018-05-30 06:55:00,061-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [5c2da757] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 76a02f75
>2018-05-30 06:55:00,063-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [5c2da757] FINISH, SetHaMaintenanceModeVDSCommand, log id: 76a02f75
>2018-05-30 06:55:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c2da757] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 06:55:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c2da757] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 06:55:00,253-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6b835725] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 29d08b2
>2018-05-30 06:55:00,255-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6b835725] FINISH, GetHardwareInfoVDSCommand, log id: 29d08b2
>2018-05-30 06:55:00,267-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [4234c60d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:55:00,271-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [6a0f8f11] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:55:00,277-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7c50c59d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:55:00,288-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7c50c59d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 10970781
>2018-05-30 06:55:00,662-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7c50c59d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:55:00,662-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7c50c59d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 10970781
>2018-05-30 06:55:00,666-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7c50c59d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:55:00,667-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7c50c59d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:55:00,671-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7c50c59d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:55:00,671-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7c50c59d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:55:00,675-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7c50c59d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:55:00,676-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7c50c59d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:55:00,679-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7c50c59d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:55:00,681-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7c50c59d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7c50c59d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:55:00,688-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: c3b8d6
>2018-05-30 06:55:00,688-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:55:00,787-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:55:00,787-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:55:00,787-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:55:00,787-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:55:00,787-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: c3b8d6
>2018-05-30 06:55:00,787-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:55:00,800-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [7cf5adcc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:55:00,802-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7cf5adcc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b0a34f
>2018-05-30 06:55:00,804-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7cf5adcc] FINISH, SetVdsStatusVDSCommand, log id: b0a34f
>2018-05-30 06:55:00,836-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7cf5adcc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7cf5adcc, Job ID: 0da1590c-8147-46e8-8485-db587d6eb328, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:55:00,841-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7cf5adcc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6a0f8f11, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:55:00,843-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7cf5adcc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6a0f8f11, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:55:00,854-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [38988f69] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 06:55:00,855-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [38988f69] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 06:55:02,215-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [602cac2e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 58b98ac
>2018-05-30 06:55:02,217-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [602cac2e] FINISH, GetHardwareInfoVDSCommand, log id: 58b98ac
>2018-05-30 06:55:02,229-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [18fc1076] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:55:02,234-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [49653a5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:55:02,239-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7fe2fb4a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 06:55:02,302-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7fe2fb4a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 24fcfcfd
>2018-05-30 06:55:02,686-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7fe2fb4a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 06:55:02,686-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7fe2fb4a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 24fcfcfd
>2018-05-30 06:55:02,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7fe2fb4a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:55:02,691-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7fe2fb4a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:55:02,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7fe2fb4a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:55:02,696-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7fe2fb4a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:55:02,699-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7fe2fb4a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 06:55:02,700-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7fe2fb4a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 06:55:02,703-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7fe2fb4a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 06:55:02,705-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7fe2fb4a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7fe2fb4a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 06:55:02,712-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5452bfa8
>2018-05-30 06:55:02,712-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 06:55:02,810-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:55:02,810-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 06:55:02,810-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 06:55:02,810-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 06:55:02,810-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5452bfa8
>2018-05-30 06:55:02,811-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 06:55:02,824-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [1f31e14a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:55:02,825-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1f31e14a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3357907f
>2018-05-30 06:55:02,827-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1f31e14a] FINISH, SetVdsStatusVDSCommand, log id: 3357907f
>2018-05-30 06:55:02,836-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1f31e14a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1f31e14a, Job ID: 3c374413-1cb8-402f-b0aa-620e69580298, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 06:55:02,840-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1f31e14a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 49653a5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 06:55:02,842-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1f31e14a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 49653a5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 06:55:02,853-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [9cb3562] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 06:55:02,854-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [9cb3562] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [6edc22b5] Autorecovering 2 hosts
>2018-05-30 07:00:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [6edc22b5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:00:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [67198037] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:00:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [67198037] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:00:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [67198037] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:00:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [67198037] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:00:00,054-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [67198037] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 73d5e6d
>2018-05-30 07:00:00,057-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [67198037] FINISH, SetVdsStatusVDSCommand, log id: 73d5e6d
>2018-05-30 07:00:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [67198037] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:00:00,059-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [67198037] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:00:00,059-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [67198037] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:00:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [78f48851] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:00:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [78f48851] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:00:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [78f48851] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:00:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [78f48851] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:00:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [78f48851] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 503376ce
>2018-05-30 07:00:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [78f48851] FINISH, SetVdsStatusVDSCommand, log id: 503376ce
>2018-05-30 07:00:00,078-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [78f48851] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 61911e0a
>2018-05-30 07:00:00,080-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [78f48851] FINISH, SetHaMaintenanceModeVDSCommand, log id: 61911e0a
>2018-05-30 07:00:00,080-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [78f48851] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:00:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [78f48851] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:00:00,500-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [2ed85cad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 413de36a
>2018-05-30 07:00:00,502-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [2ed85cad] FINISH, GetHardwareInfoVDSCommand, log id: 413de36a
>2018-05-30 07:00:00,514-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [593f3f9f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:00:00,522-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [34c78a72] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:00:00,528-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5d8ae3bb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:00:00,535-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5d8ae3bb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5773a4b5
>2018-05-30 07:00:00,908-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d8ae3bb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:00:00,908-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5d8ae3bb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5773a4b5
>2018-05-30 07:00:00,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d8ae3bb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:00:00,912-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5d8ae3bb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:00:00,916-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d8ae3bb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:00:00,916-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5d8ae3bb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:00:00,920-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d8ae3bb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:00:00,921-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5d8ae3bb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:00:00,924-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5d8ae3bb] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:00:00,927-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d8ae3bb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5d8ae3bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:00:00,930-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6ab0468a
>2018-05-30 07:00:00,930-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:00:01,024-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:00:01,024-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:00:01,024-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:00:01,024-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:00:01,024-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6ab0468a
>2018-05-30 07:00:01,024-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:00:01,037-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [2e062175] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:00:01,038-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2e062175] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6b4e83c
>2018-05-30 07:00:01,040-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2e062175] FINISH, SetVdsStatusVDSCommand, log id: 6b4e83c
>2018-05-30 07:00:01,069-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e062175] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2e062175, Job ID: 0ed01f50-8e45-4436-9625-ae88080d2e82, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:00:01,073-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e062175] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 34c78a72, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:00:01,075-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e062175] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 34c78a72, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:00:01,086-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [33d3cb2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:00:01,087-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [33d3cb2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:00:01,573-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [39edb12c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 358db0b6
>2018-05-30 07:00:01,574-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [39edb12c] FINISH, GetHardwareInfoVDSCommand, log id: 358db0b6
>2018-05-30 07:00:01,586-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [425e43a9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:00:01,591-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [5c9881ef] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:00:01,596-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2c0182ef] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:00:01,602-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2c0182ef] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 592c108f
>2018-05-30 07:00:01,971-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2c0182ef] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:00:01,971-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2c0182ef] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 592c108f
>2018-05-30 07:00:01,975-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2c0182ef] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:00:01,975-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2c0182ef] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:00:01,979-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2c0182ef] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:00:01,980-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2c0182ef] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:00:01,984-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2c0182ef] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:00:01,984-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2c0182ef] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:00:01,988-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2c0182ef] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:00:01,990-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2c0182ef] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2c0182ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:00:01,993-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 66796e2d
>2018-05-30 07:00:01,993-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:00:02,089-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:00:02,089-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:00:02,089-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:00:02,089-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:00:02,089-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 66796e2d
>2018-05-30 07:00:02,089-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:00:02,101-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [63535dad] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:00:02,102-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [63535dad] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33c8c748
>2018-05-30 07:00:02,104-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [63535dad] FINISH, SetVdsStatusVDSCommand, log id: 33c8c748
>2018-05-30 07:00:02,110-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [63535dad] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 63535dad, Job ID: 815328b8-a282-42c1-944b-26012c09ee90, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:00:02,114-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [63535dad] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5c9881ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:00:02,116-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [63535dad] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5c9881ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:00:02,126-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [b6f517b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:00:02,127-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [b6f517b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:05:00,006-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [7076586f] Autorecovering 2 hosts
>2018-05-30 07:05:00,006-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [7076586f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:05:00,032-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2d0fb28a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:05:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2d0fb28a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:05:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2d0fb28a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:05:00,039-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2d0fb28a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:05:00,040-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2d0fb28a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3c62b900
>2018-05-30 07:05:00,043-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2d0fb28a] FINISH, SetVdsStatusVDSCommand, log id: 3c62b900
>2018-05-30 07:05:00,044-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2d0fb28a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:05:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2d0fb28a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:05:00,046-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [2d0fb28a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:05:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4472318a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:05:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4472318a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:05:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4472318a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:05:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4472318a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:05:00,059-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4472318a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 21fbed0f
>2018-05-30 07:05:00,062-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4472318a] FINISH, SetVdsStatusVDSCommand, log id: 21fbed0f
>2018-05-30 07:05:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [4472318a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2ac15420
>2018-05-30 07:05:00,066-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [4472318a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2ac15420
>2018-05-30 07:05:00,066-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4472318a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:05:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4472318a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:05:01,679-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [53a249ac] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4ecc368a
>2018-05-30 07:05:01,681-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [53a249ac] FINISH, GetHardwareInfoVDSCommand, log id: 4ecc368a
>2018-05-30 07:05:01,695-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3081f566] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:05:01,704-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [160d900e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:05:01,709-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [21193063] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:05:01,716-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [21193063] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 42c3b44f
>2018-05-30 07:05:02,106-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21193063] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:05:02,106-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [21193063] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 42c3b44f
>2018-05-30 07:05:02,110-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21193063] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:05:02,111-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [21193063] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:05:02,115-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21193063] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:05:02,116-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [21193063] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:05:02,120-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21193063] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:05:02,120-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [21193063] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:05:02,124-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [21193063] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:05:02,127-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21193063] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 21193063, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:05:02,130-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 23bb91b8
>2018-05-30 07:05:02,130-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:05:02,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:05:02,228-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:05:02,228-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:05:02,228-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:05:02,229-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 23bb91b8
>2018-05-30 07:05:02,229-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:05:02,243-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [560ce9f3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:05:02,244-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [560ce9f3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c13f5cb
>2018-05-30 07:05:02,247-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [560ce9f3] FINISH, SetVdsStatusVDSCommand, log id: 7c13f5cb
>2018-05-30 07:05:02,252-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [560ce9f3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 560ce9f3, Job ID: 3653c4ba-e4d5-4c41-a789-7457289e7009, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:05:02,257-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [560ce9f3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 160d900e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:05:02,259-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [560ce9f3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 160d900e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:05:02,270-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [49a8adf] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:05:02,271-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [49a8adf] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:05:02,808-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5e603e94] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6e9177ea
>2018-05-30 07:05:02,810-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5e603e94] FINISH, GetHardwareInfoVDSCommand, log id: 6e9177ea
>2018-05-30 07:05:02,823-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [2499f555] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:05:02,827-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [1159aee3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:05:02,832-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [3c97ee73] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:05:02,840-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [3c97ee73] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3b9d0942
>2018-05-30 07:05:03,214-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c97ee73] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:05:03,214-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [3c97ee73] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3b9d0942
>2018-05-30 07:05:03,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c97ee73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:05:03,219-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [3c97ee73] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:05:03,223-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c97ee73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:05:03,223-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [3c97ee73] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:05:03,227-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c97ee73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:05:03,228-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [3c97ee73] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:05:03,230-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [3c97ee73] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:05:03,233-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c97ee73] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3c97ee73, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:05:03,236-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 171c5e4c
>2018-05-30 07:05:03,236-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:05:03,338-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:05:03,338-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:05:03,338-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:05:03,338-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:05:03,338-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 171c5e4c
>2018-05-30 07:05:03,338-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:05:03,351-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1101fcc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:05:03,352-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1101fcc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 66e8d1f6
>2018-05-30 07:05:03,354-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1101fcc] FINISH, SetVdsStatusVDSCommand, log id: 66e8d1f6
>2018-05-30 07:05:03,359-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1101fcc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1101fcc, Job ID: 85cd10e7-c40e-43b9-b4dc-55bf9e49cfc3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:05:03,364-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1101fcc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1159aee3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:05:03,366-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1101fcc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1159aee3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:05:03,376-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [61ea8921] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:05:03,377-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [61ea8921] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [5fa7e7e9] Autorecovering 2 hosts
>2018-05-30 07:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [5fa7e7e9] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:10:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [19fc9253] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:10:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [19fc9253] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:10:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [19fc9253] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:10:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [19fc9253] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:10:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [19fc9253] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 57c9f1e
>2018-05-30 07:10:00,076-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [19fc9253] FINISH, SetVdsStatusVDSCommand, log id: 57c9f1e
>2018-05-30 07:10:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [19fc9253] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:10:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [19fc9253] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:10:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [19fc9253] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:10:00,085-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6d86f3c9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:10:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6d86f3c9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:10:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6d86f3c9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:10:00,091-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6d86f3c9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:10:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6d86f3c9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5ef8ec15
>2018-05-30 07:10:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6d86f3c9] FINISH, SetVdsStatusVDSCommand, log id: 5ef8ec15
>2018-05-30 07:10:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [6d86f3c9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1784a3fe
>2018-05-30 07:10:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [6d86f3c9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1784a3fe
>2018-05-30 07:10:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6d86f3c9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:10:00,100-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6d86f3c9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:10:01,129-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3e5f75bd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 14ba53b5
>2018-05-30 07:10:01,131-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3e5f75bd] FINISH, GetHardwareInfoVDSCommand, log id: 14ba53b5
>2018-05-30 07:10:01,144-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [53eba1dd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:10:01,153-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [6451c212] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:10:01,158-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [537c4aa9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:10:01,218-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [537c4aa9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7ba37cfe
>2018-05-30 07:10:01,587-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [537c4aa9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:10:01,587-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [537c4aa9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7ba37cfe
>2018-05-30 07:10:01,591-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [537c4aa9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:10:01,592-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [537c4aa9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:10:01,596-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [537c4aa9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:10:01,596-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [537c4aa9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:10:01,600-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [537c4aa9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:10:01,601-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [537c4aa9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:10:01,603-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [537c4aa9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:10:01,606-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [537c4aa9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 537c4aa9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:10:01,613-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4088a1cb
>2018-05-30 07:10:01,613-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:10:01,710-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:10:01,710-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:10:01,710-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:10:01,710-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:10:01,710-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4088a1cb
>2018-05-30 07:10:01,710-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:10:01,738-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [628ef096] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:10:01,739-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [628ef096] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 21273010
>2018-05-30 07:10:01,741-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [628ef096] FINISH, SetVdsStatusVDSCommand, log id: 21273010
>2018-05-30 07:10:01,773-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [628ef096] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 628ef096, Job ID: 13bcc270-0a11-4aac-b2ba-9d12d00b3e46, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:10:01,778-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [628ef096] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6451c212, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:10:01,780-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [628ef096] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6451c212, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:10:01,791-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [4b9a1af] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:10:01,792-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [4b9a1af] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:10:02,895-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [46de57d5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4e8d85ec
>2018-05-30 07:10:02,897-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [46de57d5] FINISH, GetHardwareInfoVDSCommand, log id: 4e8d85ec
>2018-05-30 07:10:02,911-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [5a7e996c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:10:02,919-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [1a0000b7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:10:02,925-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3e2ad266] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:10:02,936-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3e2ad266] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1c2fc4e2
>2018-05-30 07:10:03,313-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e2ad266] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:10:03,313-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3e2ad266] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1c2fc4e2
>2018-05-30 07:10:03,317-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e2ad266] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:10:03,317-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e2ad266] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:10:03,322-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e2ad266] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:10:03,322-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e2ad266] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:10:03,326-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e2ad266] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:10:03,327-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e2ad266] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:10:03,329-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3e2ad266] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:10:03,332-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e2ad266] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3e2ad266, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:10:03,335-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7d26a7d2
>2018-05-30 07:10:03,335-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:10:03,429-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:10:03,429-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:10:03,429-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:10:03,429-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:10:03,429-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7d26a7d2
>2018-05-30 07:10:03,429-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:10:03,442-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [2ced2cbc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:10:03,443-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2ced2cbc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 147fe90a
>2018-05-30 07:10:03,445-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2ced2cbc] FINISH, SetVdsStatusVDSCommand, log id: 147fe90a
>2018-05-30 07:10:03,451-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2ced2cbc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2ced2cbc, Job ID: f2db7382-857f-4885-b350-6d461a370c1a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:10:03,455-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2ced2cbc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1a0000b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:10:03,457-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2ced2cbc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1a0000b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:10:03,468-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [34577400] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:10:03,469-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [34577400] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:15:00,015-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [548cd5d6] Autorecovering 2 hosts
>2018-05-30 07:15:00,015-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [548cd5d6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:15:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78051c4d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:15:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78051c4d] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:15:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78051c4d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:15:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78051c4d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:15:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78051c4d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2bc5825d
>2018-05-30 07:15:00,081-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78051c4d] FINISH, SetVdsStatusVDSCommand, log id: 2bc5825d
>2018-05-30 07:15:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78051c4d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:15:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78051c4d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:15:00,084-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [78051c4d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:15:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [29143a6c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:15:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [29143a6c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:15:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [29143a6c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:15:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [29143a6c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:15:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [29143a6c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5308d2ab
>2018-05-30 07:15:00,100-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [29143a6c] FINISH, SetVdsStatusVDSCommand, log id: 5308d2ab
>2018-05-30 07:15:00,102-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [29143a6c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 61c36a36
>2018-05-30 07:15:00,105-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [29143a6c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 61c36a36
>2018-05-30 07:15:00,105-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [29143a6c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:15:00,106-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [29143a6c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:15:01,042-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2119db7e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 15a65cd6
>2018-05-30 07:15:01,044-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2119db7e] FINISH, GetHardwareInfoVDSCommand, log id: 15a65cd6
>2018-05-30 07:15:01,057-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [431422b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:15:01,066-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [773ca1d9] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:15:01,071-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [67423b6d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:15:01,083-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [67423b6d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 21248cd1
>2018-05-30 07:15:01,461-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [67423b6d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:15:01,461-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [67423b6d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 21248cd1
>2018-05-30 07:15:01,465-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [67423b6d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:15:01,466-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [67423b6d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:15:01,469-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [67423b6d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:15:01,470-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [67423b6d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:15:01,474-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [67423b6d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:15:01,474-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [67423b6d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:15:01,477-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [67423b6d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:15:01,480-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [67423b6d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 67423b6d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:15:01,486-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 75eafd7d
>2018-05-30 07:15:01,486-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:15:01,584-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:15:01,584-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:15:01,584-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:15:01,584-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:15:01,584-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 75eafd7d
>2018-05-30 07:15:01,584-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:15:01,597-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [1675ab2f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:15:01,598-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1675ab2f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f98bbe7
>2018-05-30 07:15:01,600-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1675ab2f] FINISH, SetVdsStatusVDSCommand, log id: 5f98bbe7
>2018-05-30 07:15:01,631-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1675ab2f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1675ab2f, Job ID: 8ba00a2a-f2e7-49d6-8ac2-577d4f3117c6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:15:01,636-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1675ab2f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 773ca1d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:15:01,638-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1675ab2f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 773ca1d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:15:01,648-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [19869c1d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:15:01,650-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [19869c1d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:15:02,492-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [488493bb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 194bdb1d
>2018-05-30 07:15:02,494-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [488493bb] FINISH, GetHardwareInfoVDSCommand, log id: 194bdb1d
>2018-05-30 07:15:02,507-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [58e350c6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:15:02,515-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5ab01986] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:15:02,521-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [311ece45] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:15:02,527-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [311ece45] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 58bc0b33
>2018-05-30 07:15:02,903-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [311ece45] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:15:02,903-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [311ece45] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 58bc0b33
>2018-05-30 07:15:02,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [311ece45] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:15:02,908-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [311ece45] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:15:02,911-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [311ece45] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:15:02,912-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [311ece45] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:15:02,916-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [311ece45] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:15:02,917-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [311ece45] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:15:02,919-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [311ece45] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:15:02,922-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [311ece45] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 311ece45, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:15:02,925-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7af9a9a3
>2018-05-30 07:15:02,925-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:15:03,023-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:15:03,023-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:15:03,023-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:15:03,024-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:15:03,024-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7af9a9a3
>2018-05-30 07:15:03,024-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:15:03,036-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [60444e4b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:15:03,037-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [60444e4b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 457ed7c3
>2018-05-30 07:15:03,040-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [60444e4b] FINISH, SetVdsStatusVDSCommand, log id: 457ed7c3
>2018-05-30 07:15:03,045-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [60444e4b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 60444e4b, Job ID: f44be41a-ff35-447b-a190-b02a59416d33, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:15:03,049-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [60444e4b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5ab01986, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:15:03,051-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [60444e4b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5ab01986, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:15:03,062-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [668a8984] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:15:03,063-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [668a8984] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [40cdf13b] Autorecovering 2 hosts
>2018-05-30 07:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [40cdf13b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:20:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7d4bcb88] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7d4bcb88] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7d4bcb88] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:20:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7d4bcb88] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:20:00,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7d4bcb88] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 698f0bb8
>2018-05-30 07:20:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7d4bcb88] FINISH, SetVdsStatusVDSCommand, log id: 698f0bb8
>2018-05-30 07:20:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7d4bcb88] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:20:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7d4bcb88] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:20:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7d4bcb88] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:20:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b93f4dd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:20:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b93f4dd] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:20:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b93f4dd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:20:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b93f4dd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:20:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7b93f4dd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7fbed063
>2018-05-30 07:20:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7b93f4dd] FINISH, SetVdsStatusVDSCommand, log id: 7fbed063
>2018-05-30 07:20:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [7b93f4dd] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: a519257
>2018-05-30 07:20:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [7b93f4dd] FINISH, SetHaMaintenanceModeVDSCommand, log id: a519257
>2018-05-30 07:20:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b93f4dd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:20:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b93f4dd] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:20:00,782-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [d01d73b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 66d7ac9
>2018-05-30 07:20:00,784-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [d01d73b] FINISH, GetHardwareInfoVDSCommand, log id: 66d7ac9
>2018-05-30 07:20:00,796-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [31d33d3f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:20:00,801-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2fef773d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:20:00,806-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4d300865] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:20:00,818-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4d300865] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 60d8e6b4
>2018-05-30 07:20:02,273-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [309d21cb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 65f97ebb
>2018-05-30 07:20:02,274-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [309d21cb] FINISH, GetHardwareInfoVDSCommand, log id: 65f97ebb
>2018-05-30 07:20:02,287-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [420cc60a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:20:02,291-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [74069] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:20:02,296-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2f93801c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:20:02,353-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2f93801c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2d7bb419
>2018-05-30 07:20:02,733-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2f93801c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:20:02,733-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2f93801c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2d7bb419
>2018-05-30 07:20:02,737-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2f93801c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:20:02,738-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2f93801c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:20:02,741-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2f93801c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:20:02,742-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2f93801c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:20:02,745-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2f93801c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:20:02,746-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2f93801c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:20:02,749-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2f93801c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:20:02,751-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2f93801c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2f93801c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:20:02,759-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1041758
>2018-05-30 07:20:02,759-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:20:02,858-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:20:02,858-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:20:02,858-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:20:02,859-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:20:02,859-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1041758
>2018-05-30 07:20:02,859-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:20:02,871-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [4896be06] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:20:02,872-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4896be06] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 48fa2f94
>2018-05-30 07:20:02,875-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4896be06] FINISH, SetVdsStatusVDSCommand, log id: 48fa2f94
>2018-05-30 07:20:02,883-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4896be06] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4896be06, Job ID: 96345814-ae7b-4bb0-a2d2-dc0bb366a7c3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:20:02,887-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4896be06] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 74069, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:20:02,889-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4896be06] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 74069, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:20:02,900-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [41c80b5c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:20:02,901-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [41c80b5c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:20:03,138-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d300865] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:20:03,138-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4d300865] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 60d8e6b4
>2018-05-30 07:20:03,142-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d300865] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:20:03,143-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d300865] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:20:03,146-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d300865] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:20:03,147-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d300865] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:20:03,151-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d300865] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:20:03,151-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d300865] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:20:03,154-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4d300865] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:20:03,157-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d300865] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d300865, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:20:03,159-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7f133bac
>2018-05-30 07:20:03,159-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:20:03,256-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:20:03,256-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:20:03,256-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:20:03,256-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:20:03,256-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7f133bac
>2018-05-30 07:20:03,256-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:20:03,268-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [23c14c1b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:20:03,269-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [23c14c1b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33602198
>2018-05-30 07:20:03,271-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [23c14c1b] FINISH, SetVdsStatusVDSCommand, log id: 33602198
>2018-05-30 07:20:03,276-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [23c14c1b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 23c14c1b, Job ID: e7bf72de-c0a4-4705-9d35-13049f5ed7ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:20:03,281-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [23c14c1b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2fef773d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:20:03,283-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [23c14c1b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2fef773d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:20:03,293-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [2d98cdae] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:20:03,294-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [2d98cdae] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [72981ac6] Autorecovering 2 hosts
>2018-05-30 07:25:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [72981ac6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:25:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2d839db6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:25:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2d839db6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:25:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2d839db6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:25:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2d839db6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:25:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2d839db6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1eddae4d
>2018-05-30 07:25:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2d839db6] FINISH, SetVdsStatusVDSCommand, log id: 1eddae4d
>2018-05-30 07:25:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2d839db6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:25:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2d839db6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:25:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [2d839db6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:25:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d5bafcf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:25:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d5bafcf] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:25:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d5bafcf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:25:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d5bafcf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:25:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [d5bafcf] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7a846b94
>2018-05-30 07:25:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [d5bafcf] FINISH, SetVdsStatusVDSCommand, log id: 7a846b94
>2018-05-30 07:25:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [d5bafcf] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 621b164f
>2018-05-30 07:25:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [d5bafcf] FINISH, SetHaMaintenanceModeVDSCommand, log id: 621b164f
>2018-05-30 07:25:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d5bafcf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:25:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d5bafcf] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:25:00,632-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7c486e85] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 34fa973b
>2018-05-30 07:25:00,634-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7c486e85] FINISH, GetHardwareInfoVDSCommand, log id: 34fa973b
>2018-05-30 07:25:00,646-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [59904137] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:25:00,655-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [2f21b60e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:25:00,660-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [40da236a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:25:00,671-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [40da236a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 505efdd4
>2018-05-30 07:25:00,937-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [712816b4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 56cf44ca
>2018-05-30 07:25:00,939-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [712816b4] FINISH, GetHardwareInfoVDSCommand, log id: 56cf44ca
>2018-05-30 07:25:00,951-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [442f75f1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:25:00,956-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [2e157a02] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:25:00,961-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6607c63e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:25:00,967-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6607c63e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 794bf8c3
>2018-05-30 07:25:01,055-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [40da236a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:25:01,055-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [40da236a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 505efdd4
>2018-05-30 07:25:01,058-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [40da236a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:25:01,059-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [40da236a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:25:01,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [40da236a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:25:01,064-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [40da236a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:25:01,067-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [40da236a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:25:01,068-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [40da236a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:25:01,070-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [40da236a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:25:01,073-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [40da236a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 40da236a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:25:01,081-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7415724e
>2018-05-30 07:25:01,081-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:25:01,176-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:25:01,176-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:25:01,177-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:25:01,177-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:25:01,177-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7415724e
>2018-05-30 07:25:01,177-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:25:01,189-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [1b4f7735] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:25:01,190-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1b4f7735] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 69f35dce
>2018-05-30 07:25:01,192-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1b4f7735] FINISH, SetVdsStatusVDSCommand, log id: 69f35dce
>2018-05-30 07:25:01,225-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1b4f7735] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1b4f7735, Job ID: ca2bc444-b7d2-47b3-8490-285379a564be, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:25:01,229-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1b4f7735] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2f21b60e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:25:01,231-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1b4f7735] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2f21b60e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:25:01,241-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [1b91bb68] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:25:01,242-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [1b91bb68] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:25:01,345-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6607c63e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:25:01,345-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6607c63e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 794bf8c3
>2018-05-30 07:25:01,348-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6607c63e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:25:01,349-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6607c63e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:25:01,353-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6607c63e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:25:01,353-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6607c63e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:25:01,357-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6607c63e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:25:01,358-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6607c63e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:25:01,361-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6607c63e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:25:01,363-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6607c63e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6607c63e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:25:01,366-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: a1df0f2
>2018-05-30 07:25:01,366-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:25:01,465-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:25:01,465-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:25:01,465-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:25:01,465-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:25:01,465-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: a1df0f2
>2018-05-30 07:25:01,465-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:25:01,477-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [695f7f4d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:25:01,479-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [695f7f4d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4c194a7
>2018-05-30 07:25:01,481-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [695f7f4d] FINISH, SetVdsStatusVDSCommand, log id: 4c194a7
>2018-05-30 07:25:01,486-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [695f7f4d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 695f7f4d, Job ID: ab0e1405-adef-4663-91b2-946cc51e8b0a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:25:01,490-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [695f7f4d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2e157a02, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:25:01,492-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [695f7f4d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2e157a02, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:25:01,502-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [7c6e0218] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:25:01,503-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [7c6e0218] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [71cf5e96] Autorecovering 2 hosts
>2018-05-30 07:30:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [71cf5e96] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:30:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [c268f30] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:30:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [c268f30] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:30:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [c268f30] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:30:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [c268f30] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:30:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [c268f30] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ad24737
>2018-05-30 07:30:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [c268f30] FINISH, SetVdsStatusVDSCommand, log id: ad24737
>2018-05-30 07:30:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [c268f30] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:30:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [c268f30] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:30:00,078-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [c268f30] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:30:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [619f00af] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:30:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [619f00af] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:30:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [619f00af] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:30:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [619f00af] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:30:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [619f00af] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ad19550
>2018-05-30 07:30:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [619f00af] FINISH, SetVdsStatusVDSCommand, log id: 6ad19550
>2018-05-30 07:30:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [619f00af] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7328c72
>2018-05-30 07:30:00,098-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [619f00af] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7328c72
>2018-05-30 07:30:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [619f00af] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:30:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [619f00af] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:30:01,851-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6e832ca6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 730827fb
>2018-05-30 07:30:01,852-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6e832ca6] FINISH, GetHardwareInfoVDSCommand, log id: 730827fb
>2018-05-30 07:30:01,865-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [75c2d8e6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:30:01,870-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [4c905204] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:30:01,874-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [5ff6c697] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:30:01,881-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [5ff6c697] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2b52d70f
>2018-05-30 07:30:02,113-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1872b398] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1807730b
>2018-05-30 07:30:02,115-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1872b398] FINISH, GetHardwareInfoVDSCommand, log id: 1807730b
>2018-05-30 07:30:02,127-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [724c944b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:30:02,131-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [408d35a1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:30:02,136-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7afbca16] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:30:02,142-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7afbca16] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 60c3de17
>2018-05-30 07:30:02,258-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ff6c697] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:30:02,259-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [5ff6c697] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2b52d70f
>2018-05-30 07:30:02,266-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ff6c697] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:30:02,267-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5ff6c697] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:30:02,270-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ff6c697] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:30:02,271-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5ff6c697] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:30:02,275-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ff6c697] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:30:02,276-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5ff6c697] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:30:02,278-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [5ff6c697] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:30:02,281-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ff6c697] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5ff6c697, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:30:02,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 73eee22b
>2018-05-30 07:30:02,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:30:02,383-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:30:02,383-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:30:02,383-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:30:02,383-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:30:02,383-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 73eee22b
>2018-05-30 07:30:02,383-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:30:02,396-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [219cd652] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:30:02,397-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [219cd652] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53e90279
>2018-05-30 07:30:02,399-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [219cd652] FINISH, SetVdsStatusVDSCommand, log id: 53e90279
>2018-05-30 07:30:02,405-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [219cd652] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 219cd652, Job ID: 36932046-fd1d-4390-9d89-c98fd6f4009a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:30:02,410-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [219cd652] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4c905204, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:30:02,412-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [219cd652] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4c905204, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:30:02,422-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [2b9dc66c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:30:02,423-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2b9dc66c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:30:03,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7afbca16] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:30:03,508-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7afbca16] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 60c3de17
>2018-05-30 07:30:03,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7afbca16] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:30:03,512-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7afbca16] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:30:03,516-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7afbca16] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:30:03,517-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7afbca16] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:30:03,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7afbca16] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:30:03,521-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7afbca16] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:30:03,524-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7afbca16] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:30:03,527-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7afbca16] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7afbca16, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:30:03,531-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: f5b383b
>2018-05-30 07:30:03,531-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:30:03,629-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:30:03,629-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:30:03,629-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:30:03,629-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:30:03,629-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: f5b383b
>2018-05-30 07:30:03,629-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:30:03,642-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [46b79872] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:30:03,643-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [46b79872] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 57d27d0f
>2018-05-30 07:30:03,645-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [46b79872] FINISH, SetVdsStatusVDSCommand, log id: 57d27d0f
>2018-05-30 07:30:03,651-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [46b79872] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 46b79872, Job ID: e35ae683-4d3d-4911-9923-738428c1906f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:30:03,655-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [46b79872] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 408d35a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:30:03,657-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [46b79872] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 408d35a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:30:03,668-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [35f11183] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:30:03,669-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [35f11183] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:35:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [539d36e6] Autorecovering 2 hosts
>2018-05-30 07:35:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [539d36e6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:35:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73c91fb9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:35:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73c91fb9] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:35:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73c91fb9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:35:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73c91fb9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:35:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73c91fb9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 71feb9d4
>2018-05-30 07:35:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73c91fb9] FINISH, SetVdsStatusVDSCommand, log id: 71feb9d4
>2018-05-30 07:35:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73c91fb9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:35:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73c91fb9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:35:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [73c91fb9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:35:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68e33015] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:35:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68e33015] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:35:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68e33015] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:35:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68e33015] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:35:00,088-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [68e33015] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 342b39ea
>2018-05-30 07:35:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [68e33015] FINISH, SetVdsStatusVDSCommand, log id: 342b39ea
>2018-05-30 07:35:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [68e33015] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4d54958
>2018-05-30 07:35:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [68e33015] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4d54958
>2018-05-30 07:35:00,095-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68e33015] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:35:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68e33015] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:35:01,289-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7b357218] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2824c11d
>2018-05-30 07:35:01,290-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7b357218] FINISH, GetHardwareInfoVDSCommand, log id: 2824c11d
>2018-05-30 07:35:01,303-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [6eb58fa5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:35:01,312-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1baa5a3c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:35:01,317-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [57b8bf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:35:01,328-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [57b8bf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2c49cc66
>2018-05-30 07:35:01,703-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b8bf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:35:01,703-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [57b8bf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2c49cc66
>2018-05-30 07:35:01,707-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b8bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:35:01,708-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [57b8bf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:35:01,711-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b8bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:35:01,712-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [57b8bf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:35:01,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b8bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:35:01,716-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [57b8bf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:35:01,719-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [57b8bf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:35:01,722-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b8bf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 57b8bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:35:01,728-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 15f32612
>2018-05-30 07:35:01,728-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:35:01,830-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:35:01,830-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:35:01,830-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:35:01,830-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:35:01,830-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 15f32612
>2018-05-30 07:35:01,830-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:35:01,844-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [28a6480b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:35:01,845-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [28a6480b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33e54af2
>2018-05-30 07:35:01,847-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [28a6480b] FINISH, SetVdsStatusVDSCommand, log id: 33e54af2
>2018-05-30 07:35:01,878-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [28a6480b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 28a6480b, Job ID: c768bdeb-99d1-4759-8f57-a16bea8d9cea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:35:01,883-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [28a6480b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1baa5a3c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:35:01,884-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [28a6480b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1baa5a3c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:35:01,895-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [73a91267] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:35:01,896-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [73a91267] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:35:03,069-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [1c448d1a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5edad36f
>2018-05-30 07:35:03,071-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [1c448d1a] FINISH, GetHardwareInfoVDSCommand, log id: 5edad36f
>2018-05-30 07:35:03,083-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [4e62389a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:35:03,092-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [17e95d8a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:35:03,096-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [507346d5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:35:03,108-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [507346d5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 21f35cc9
>2018-05-30 07:35:03,493-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [507346d5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:35:03,493-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [507346d5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 21f35cc9
>2018-05-30 07:35:03,497-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [507346d5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:35:03,497-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [507346d5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:35:03,501-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [507346d5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:35:03,502-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [507346d5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:35:03,505-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [507346d5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:35:03,506-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [507346d5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:35:03,509-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [507346d5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:35:03,511-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [507346d5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 507346d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:35:03,517-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a92b76d
>2018-05-30 07:35:03,517-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:35:03,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:35:03,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:35:03,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:35:03,614-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:35:03,614-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a92b76d
>2018-05-30 07:35:03,614-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:35:03,627-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [56178bad] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:35:03,628-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [56178bad] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 719a973
>2018-05-30 07:35:03,630-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [56178bad] FINISH, SetVdsStatusVDSCommand, log id: 719a973
>2018-05-30 07:35:03,661-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [56178bad] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 56178bad, Job ID: 345e0509-69bd-48b1-9dcd-976dc18a7081, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:35:03,666-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [56178bad] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17e95d8a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:35:03,668-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [56178bad] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17e95d8a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:35:03,679-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [5a368bae] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:35:03,680-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [5a368bae] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [d570fd8] Autorecovering 2 hosts
>2018-05-30 07:40:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [d570fd8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:40:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5575fd13] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5575fd13] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5575fd13] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:40:00,070-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5575fd13] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:40:00,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5575fd13] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d727fee
>2018-05-30 07:40:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5575fd13] FINISH, SetVdsStatusVDSCommand, log id: 3d727fee
>2018-05-30 07:40:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5575fd13] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:40:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5575fd13] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:40:00,077-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [5575fd13] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:40:00,083-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [58a97e5a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:40:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [58a97e5a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:40:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [58a97e5a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:40:00,089-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [58a97e5a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:40:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [58a97e5a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 201540e1
>2018-05-30 07:40:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [58a97e5a] FINISH, SetVdsStatusVDSCommand, log id: 201540e1
>2018-05-30 07:40:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [58a97e5a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2462267a
>2018-05-30 07:40:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [58a97e5a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2462267a
>2018-05-30 07:40:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [58a97e5a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:40:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [58a97e5a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:40:01,293-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [18976593] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 17cab53
>2018-05-30 07:40:01,295-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [18976593] FINISH, GetHardwareInfoVDSCommand, log id: 17cab53
>2018-05-30 07:40:01,308-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3a038aa8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:40:01,317-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5f30c767] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:40:01,322-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [cf0f20b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:40:01,328-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [cf0f20b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 49affcb0
>2018-05-30 07:40:01,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [cf0f20b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:40:01,704-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [cf0f20b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 49affcb0
>2018-05-30 07:40:01,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [cf0f20b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:40:01,709-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [cf0f20b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:40:01,712-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [cf0f20b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:40:01,713-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [cf0f20b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:40:01,717-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [cf0f20b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:40:01,717-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [cf0f20b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:40:01,720-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [cf0f20b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:40:01,722-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [cf0f20b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: cf0f20b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:40:01,725-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2f121e2b
>2018-05-30 07:40:01,725-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:40:01,826-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:40:01,826-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:40:01,826-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:40:01,826-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:40:01,826-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2f121e2b
>2018-05-30 07:40:01,826-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:40:01,838-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [5bfa1db8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:40:01,840-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5bfa1db8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 74f705e4
>2018-05-30 07:40:01,842-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5bfa1db8] FINISH, SetVdsStatusVDSCommand, log id: 74f705e4
>2018-05-30 07:40:01,873-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5bfa1db8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5bfa1db8, Job ID: 730d48f5-203c-4455-90f5-16424860ac74, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:40:01,877-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5bfa1db8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5f30c767, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:40:01,879-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5bfa1db8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5f30c767, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:40:01,890-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [13f5eb0c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:40:01,891-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [13f5eb0c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:40:02,537-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7f111ec4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 480f7c2c
>2018-05-30 07:40:02,539-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7f111ec4] FINISH, GetHardwareInfoVDSCommand, log id: 480f7c2c
>2018-05-30 07:40:02,551-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7f5e62e2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:40:02,560-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [1948a1c7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:40:02,565-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [776dea78] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:40:02,571-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [776dea78] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1974cc84
>2018-05-30 07:40:02,936-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [776dea78] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:40:02,936-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [776dea78] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1974cc84
>2018-05-30 07:40:02,939-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [776dea78] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:40:02,940-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [776dea78] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:40:02,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [776dea78] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:40:02,945-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [776dea78] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:40:02,949-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [776dea78] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:40:02,949-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [776dea78] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:40:02,952-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [776dea78] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:40:02,954-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [776dea78] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 776dea78, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:40:02,957-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 74a0cbd6
>2018-05-30 07:40:02,957-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:40:03,053-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:40:03,053-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:40:03,053-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:40:03,053-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:40:03,053-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 74a0cbd6
>2018-05-30 07:40:03,053-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:40:03,065-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [1da0861b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:40:03,066-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1da0861b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f5d2379
>2018-05-30 07:40:03,068-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1da0861b] FINISH, SetVdsStatusVDSCommand, log id: f5d2379
>2018-05-30 07:40:03,098-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1da0861b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1da0861b, Job ID: 971be869-4806-4b01-b3da-c43be59d8c8a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:40:03,102-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1da0861b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1948a1c7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:40:03,105-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1da0861b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1948a1c7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:40:03,115-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [2e131c1c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:40:03,116-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [2e131c1c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:45:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [2a6f260] Autorecovering 2 hosts
>2018-05-30 07:45:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [2a6f260] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:45:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [23f3ce29] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [23f3ce29] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [23f3ce29] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:45:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [23f3ce29] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:45:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [23f3ce29] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49339207
>2018-05-30 07:45:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [23f3ce29] FINISH, SetVdsStatusVDSCommand, log id: 49339207
>2018-05-30 07:45:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [23f3ce29] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:45:00,076-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [23f3ce29] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:45:00,076-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [23f3ce29] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:45:00,082-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [38feac64] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [38feac64] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [38feac64] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:45:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [38feac64] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:45:00,090-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [38feac64] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2fd2f173
>2018-05-30 07:45:00,092-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [38feac64] FINISH, SetVdsStatusVDSCommand, log id: 2fd2f173
>2018-05-30 07:45:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [38feac64] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2914e1e8
>2018-05-30 07:45:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [38feac64] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2914e1e8
>2018-05-30 07:45:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [38feac64] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:45:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [38feac64] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:45:00,756-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [41b1dad9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2ed57036
>2018-05-30 07:45:00,758-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [41b1dad9] FINISH, GetHardwareInfoVDSCommand, log id: 2ed57036
>2018-05-30 07:45:00,771-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [77483654] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:45:00,780-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [6e7c96bf] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:45:00,785-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5c2d43fb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:45:00,797-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5c2d43fb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5f03dee1
>2018-05-30 07:45:01,160-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5c2d43fb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:45:01,160-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5c2d43fb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5f03dee1
>2018-05-30 07:45:01,164-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5c2d43fb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:45:01,164-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5c2d43fb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:45:01,168-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5c2d43fb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:45:01,169-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5c2d43fb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:45:01,172-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5c2d43fb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:45:01,173-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5c2d43fb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:45:01,176-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5c2d43fb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:45:01,178-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5c2d43fb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5c2d43fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:45:01,185-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3e4b8be4
>2018-05-30 07:45:01,185-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:45:01,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:45:01,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:45:01,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:45:01,280-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:45:01,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3e4b8be4
>2018-05-30 07:45:01,280-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:45:01,293-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [6573cea1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:45:01,294-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6573cea1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d0df87f
>2018-05-30 07:45:01,296-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6573cea1] FINISH, SetVdsStatusVDSCommand, log id: 2d0df87f
>2018-05-30 07:45:01,328-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6573cea1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6573cea1, Job ID: 1859bb7d-ab65-417f-aeb4-7aa11ae83275, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:45:01,332-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6573cea1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6e7c96bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:45:01,334-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6573cea1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6e7c96bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:45:01,345-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [2154c7f8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:45:01,346-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [2154c7f8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:45:02,532-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [dc037b0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3bdb1cdb
>2018-05-30 07:45:02,534-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [dc037b0] FINISH, GetHardwareInfoVDSCommand, log id: 3bdb1cdb
>2018-05-30 07:45:02,547-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [3772ac4b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:45:02,555-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [635c843f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:45:02,561-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5d606fb5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:45:02,626-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5d606fb5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1a9c8ef8
>2018-05-30 07:45:03,009-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5d606fb5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:45:03,009-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5d606fb5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1a9c8ef8
>2018-05-30 07:45:03,013-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5d606fb5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:45:03,013-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5d606fb5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:45:03,017-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5d606fb5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:45:03,018-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5d606fb5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:45:03,021-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5d606fb5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:45:03,022-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5d606fb5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:45:03,025-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5d606fb5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:45:03,027-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5d606fb5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5d606fb5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:45:03,034-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 17e83dec
>2018-05-30 07:45:03,034-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:45:03,129-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:45:03,129-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:45:03,129-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:45:03,129-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:45:03,129-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 17e83dec
>2018-05-30 07:45:03,129-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:45:03,142-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [31c9b3f8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:45:03,143-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [31c9b3f8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42350d3
>2018-05-30 07:45:03,145-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [31c9b3f8] FINISH, SetVdsStatusVDSCommand, log id: 42350d3
>2018-05-30 07:45:03,154-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [31c9b3f8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 31c9b3f8, Job ID: c12707fc-9153-4e46-bf4d-0c264264dc66, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:45:03,158-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [31c9b3f8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 635c843f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:45:03,160-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [31c9b3f8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 635c843f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:45:03,171-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [1484b3e6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:45:03,173-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1484b3e6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [33a50a75] Autorecovering 2 hosts
>2018-05-30 07:50:00,008-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [33a50a75] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:50:00,061-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a76a383] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a76a383] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a76a383] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:50:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a76a383] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:50:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7a76a383] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32033ed
>2018-05-30 07:50:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7a76a383] FINISH, SetVdsStatusVDSCommand, log id: 32033ed
>2018-05-30 07:50:00,074-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a76a383] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:50:00,075-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a76a383] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:50:00,075-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [7a76a383] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:50:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d6a911c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d6a911c] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d6a911c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:50:00,088-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d6a911c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:50:00,089-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3d6a911c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3cc7c6e1
>2018-05-30 07:50:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3d6a911c] FINISH, SetVdsStatusVDSCommand, log id: 3cc7c6e1
>2018-05-30 07:50:00,094-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [3d6a911c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 155bbb8a
>2018-05-30 07:50:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [3d6a911c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 155bbb8a
>2018-05-30 07:50:00,096-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d6a911c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:50:00,097-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d6a911c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:50:00,802-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [5b77834e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1e540e89
>2018-05-30 07:50:00,804-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [5b77834e] FINISH, GetHardwareInfoVDSCommand, log id: 1e540e89
>2018-05-30 07:50:00,817-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2c079313] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:50:00,825-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [14acacfa] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:50:00,830-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [5dd4796e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:50:00,842-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [5dd4796e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: e0c9aa7
>2018-05-30 07:50:01,217-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5dd4796e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:50:01,217-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [5dd4796e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: e0c9aa7
>2018-05-30 07:50:01,221-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5dd4796e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:50:01,222-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5dd4796e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:50:01,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5dd4796e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:50:01,227-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5dd4796e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:50:01,231-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5dd4796e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:50:01,231-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5dd4796e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:50:01,234-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [5dd4796e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:50:01,237-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5dd4796e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5dd4796e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:50:01,243-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: c675911
>2018-05-30 07:50:01,243-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:50:01,341-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:50:01,341-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:50:01,341-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:50:01,341-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:50:01,341-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: c675911
>2018-05-30 07:50:01,341-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:50:01,354-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [4a41a996] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:50:01,355-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4a41a996] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2586ac70
>2018-05-30 07:50:01,357-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4a41a996] FINISH, SetVdsStatusVDSCommand, log id: 2586ac70
>2018-05-30 07:50:01,389-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4a41a996] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4a41a996, Job ID: c10070a5-fc97-4838-b208-bfb59696d3a5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:50:01,394-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4a41a996] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 14acacfa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:50:01,396-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4a41a996] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 14acacfa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:50:01,407-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [1ac9ea8f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:50:01,408-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [1ac9ea8f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:50:01,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7176105e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 66019261
>2018-05-30 07:50:01,999-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7176105e] FINISH, GetHardwareInfoVDSCommand, log id: 66019261
>2018-05-30 07:50:02,012-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [48b85720] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:50:02,016-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [43d37e16] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:50:02,021-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6e1ce468] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:50:02,028-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6e1ce468] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 673f8840
>2018-05-30 07:50:03,406-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6e1ce468] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:50:03,406-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6e1ce468] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 673f8840
>2018-05-30 07:50:03,410-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6e1ce468] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:50:03,411-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6e1ce468] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:50:03,415-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6e1ce468] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:50:03,415-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6e1ce468] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:50:03,419-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6e1ce468] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:50:03,420-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6e1ce468] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:50:03,423-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6e1ce468] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:50:03,426-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6e1ce468] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6e1ce468, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:50:03,432-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 206d7877
>2018-05-30 07:50:03,432-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:50:03,532-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:50:03,532-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:50:03,532-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:50:03,532-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:50:03,532-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 206d7877
>2018-05-30 07:50:03,533-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:50:03,546-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [742a7045] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:50:03,547-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [742a7045] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 66172242
>2018-05-30 07:50:03,550-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [742a7045] FINISH, SetVdsStatusVDSCommand, log id: 66172242
>2018-05-30 07:50:03,581-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [742a7045] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 742a7045, Job ID: 907c9255-6b54-4773-b49a-21ce327d6dd5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:50:03,586-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [742a7045] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 43d37e16, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:50:03,588-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [742a7045] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 43d37e16, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:50:03,599-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [4f4642c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:50:03,600-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [4f4642c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:54:38,388-04 INFO  [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler10) [52b93cf5] Backup check started.
>2018-05-30 07:54:38,402-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52b93cf5] EVENT_ID: ENGINE_NO_FULL_BACKUP(9,022), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: There is no full backup available, please run engine-backup to prevent data loss in case of corruption.
>2018-05-30 07:54:38,402-04 INFO  [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler10) [52b93cf5] Backup check completed.
>2018-05-30 07:54:43,581-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler8) [5d51fdd7] Attempting to update VMs/Templates Ovf.
>2018-05-30 07:54:43,583-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3084667b] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 07:54:43,584-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3084667b] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 07:54:43,584-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3084667b] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:54:43,634-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3084667b] Attempting to update VM OVFs in Data Center 'Default'
>2018-05-30 07:54:43,638-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3084667b] Successfully updated VM OVFs in Data Center 'Default'
>2018-05-30 07:54:43,638-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3084667b] Attempting to update template OVFs in Data Center 'Default'
>2018-05-30 07:54:43,638-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3084667b] Successfully updated templates OVFs in Data Center 'Default'
>2018-05-30 07:54:43,638-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3084667b] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 07:54:43,639-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3084667b] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
>2018-05-30 07:54:43,640-04 INFO  [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3084667b] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}'
>2018-05-30 07:55:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3dd8dd87] Autorecovering 2 hosts
>2018-05-30 07:55:00,004-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3dd8dd87] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:55:00,028-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c9f98e6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:55:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c9f98e6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:55:00,034-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c9f98e6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:55:00,035-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c9f98e6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:55:00,036-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [c9f98e6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 27e9de5a
>2018-05-30 07:55:00,039-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [c9f98e6] FINISH, SetVdsStatusVDSCommand, log id: 27e9de5a
>2018-05-30 07:55:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c9f98e6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:55:00,042-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c9f98e6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 07:55:00,042-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [c9f98e6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:55:00,048-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4c4d4433] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:55:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4c4d4433] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 07:55:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4c4d4433] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:55:00,054-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4c4d4433] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:55:00,055-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4c4d4433] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52714520
>2018-05-30 07:55:00,057-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4c4d4433] FINISH, SetVdsStatusVDSCommand, log id: 52714520
>2018-05-30 07:55:00,060-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4c4d4433] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5a8c16ce
>2018-05-30 07:55:00,062-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4c4d4433] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5a8c16ce
>2018-05-30 07:55:00,062-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4c4d4433] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 07:55:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4c4d4433] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 07:55:01,279-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [406740a4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 62ffd587
>2018-05-30 07:55:01,280-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [406740a4] FINISH, GetHardwareInfoVDSCommand, log id: 62ffd587
>2018-05-30 07:55:01,293-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [2a46883b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:55:01,297-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [220ab05d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:55:01,302-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [344cefb2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:55:01,309-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [344cefb2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3c6f667f
>2018-05-30 07:55:01,678-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [344cefb2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:55:01,678-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [344cefb2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3c6f667f
>2018-05-30 07:55:01,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [344cefb2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:55:01,683-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [344cefb2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:55:01,686-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [344cefb2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:55:01,687-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [344cefb2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:55:01,691-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [344cefb2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:55:01,691-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [344cefb2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:55:01,694-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [344cefb2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:55:01,697-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [344cefb2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 344cefb2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:55:01,699-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 104e2fcf
>2018-05-30 07:55:01,699-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:55:01,796-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:55:01,796-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:55:01,796-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:55:01,796-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:55:01,796-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 104e2fcf
>2018-05-30 07:55:01,796-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:55:01,809-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [2a6016cc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:55:01,810-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2a6016cc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 429e637e
>2018-05-30 07:55:01,812-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2a6016cc] FINISH, SetVdsStatusVDSCommand, log id: 429e637e
>2018-05-30 07:55:01,818-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2a6016cc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2a6016cc, Job ID: e09a2c06-6201-4511-9a2a-2a52ce8379a4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:55:01,822-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2a6016cc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 220ab05d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:55:01,824-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2a6016cc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 220ab05d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:55:01,835-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [774a9299] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 07:55:01,836-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [774a9299] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 07:55:02,133-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [5ab030be] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3366cc13
>2018-05-30 07:55:02,135-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [5ab030be] FINISH, GetHardwareInfoVDSCommand, log id: 3366cc13
>2018-05-30 07:55:02,147-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [647238a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:55:02,155-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5c451d6c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:55:02,161-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [4b4100fe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 07:55:02,172-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [4b4100fe] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7a07052d
>2018-05-30 07:55:02,553-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4b4100fe] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 07:55:02,553-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [4b4100fe] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7a07052d
>2018-05-30 07:55:02,557-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4b4100fe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:55:02,557-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4b4100fe] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:55:02,561-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4b4100fe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:55:02,562-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4b4100fe] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:55:02,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4b4100fe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 07:55:02,566-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4b4100fe] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 07:55:02,569-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [4b4100fe] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 07:55:02,572-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4b4100fe] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4b4100fe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 07:55:02,575-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6fe32c7f
>2018-05-30 07:55:02,575-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 07:55:02,673-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:55:02,673-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 07:55:02,673-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 07:55:02,673-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 07:55:02,673-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6fe32c7f
>2018-05-30 07:55:02,673-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 07:55:02,687-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [42850670] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:55:02,688-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [42850670] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a762786
>2018-05-30 07:55:02,690-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [42850670] FINISH, SetVdsStatusVDSCommand, log id: 5a762786
>2018-05-30 07:55:02,695-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42850670] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 42850670, Job ID: e1a859cf-d404-482c-961b-6d19fac32af9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 07:55:02,700-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42850670] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5c451d6c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 07:55:02,702-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42850670] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5c451d6c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 07:55:02,713-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6eae113c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 07:55:02,714-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6eae113c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 08:00:00,015-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [bccf89] Autorecovering 2 hosts
>2018-05-30 08:00:00,015-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [bccf89] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:00:00,046-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [604dc97] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 08:00:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [604dc97] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 08:00:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [604dc97] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:00:00,053-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [604dc97] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:00:00,055-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [604dc97] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f30146e
>2018-05-30 08:00:01,260-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [604dc97] FINISH, SetVdsStatusVDSCommand, log id: f30146e
>2018-05-30 08:00:01,261-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [604dc97] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:00:01,263-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [604dc97] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 08:00:01,263-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [604dc97] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:00:01,269-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6ff8808f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 08:00:01,275-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6ff8808f] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 08:00:01,276-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6ff8808f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:00:01,276-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6ff8808f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:00:01,277-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6ff8808f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 58083a3a
>2018-05-30 08:00:01,280-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6ff8808f] FINISH, SetVdsStatusVDSCommand, log id: 58083a3a
>2018-05-30 08:00:01,282-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [6ff8808f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4f1f448f
>2018-05-30 08:00:01,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [6ff8808f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4f1f448f
>2018-05-30 08:00:01,285-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6ff8808f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:00:01,286-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6ff8808f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 08:00:02,478-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [33cdac] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7e32990c
>2018-05-30 08:00:02,480-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [33cdac] FINISH, GetHardwareInfoVDSCommand, log id: 7e32990c
>2018-05-30 08:00:02,492-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [709b8846] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:00:02,497-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [78bda7b2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:00:02,502-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [1cea2794] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:00:02,513-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [1cea2794] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3cc24d62
>2018-05-30 08:00:03,357-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [74cfe4a4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 767238d8
>2018-05-30 08:00:03,358-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [74cfe4a4] FINISH, GetHardwareInfoVDSCommand, log id: 767238d8
>2018-05-30 08:00:03,371-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3621f871] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:00:03,376-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [31a7de72] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:00:03,381-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6d657845] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:00:03,388-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6d657845] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 359e9bcb
>2018-05-30 08:00:03,770-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d657845] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 08:00:03,770-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6d657845] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 359e9bcb
>2018-05-30 08:00:03,774-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d657845] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:00:03,775-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6d657845] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:00:03,778-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d657845] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:00:03,779-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6d657845] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:00:03,783-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d657845] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:00:03,784-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6d657845] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:00:03,786-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6d657845] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 08:00:03,789-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d657845] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6d657845, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 08:00:03,795-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1ad96af0
>2018-05-30 08:00:03,795-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 08:00:03,877-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1cea2794] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 08:00:03,877-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [1cea2794] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3cc24d62
>2018-05-30 08:00:03,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1cea2794] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:00:03,882-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1cea2794] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:00:03,885-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1cea2794] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:00:03,886-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1cea2794] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:00:03,890-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1cea2794] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:00:03,891-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1cea2794] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:00:03,891-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:00:03,891-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 08:00:03,891-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:00:03,891-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:00:03,891-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1ad96af0
>2018-05-30 08:00:03,891-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 08:00:03,905-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [65d4330c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:00:03,906-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [65d4330c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2b4aeb28
>2018-05-30 08:00:03,908-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [65d4330c] FINISH, SetVdsStatusVDSCommand, log id: 2b4aeb28
>2018-05-30 08:00:03,917-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [65d4330c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 65d4330c, Job ID: 1ec0154e-ad96-44a5-a8b0-30c41b5b359c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 08:00:03,921-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [65d4330c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 31a7de72, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 08:00:03,923-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [65d4330c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 31a7de72, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 08:00:03,935-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [13d6e21] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:00:03,936-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [13d6e21] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 08:00:03,941-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [1cea2794] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 08:00:03,944-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1cea2794] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1cea2794, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 08:00:03,951-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3a45613d
>2018-05-30 08:00:03,951-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 08:00:04,054-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:00:04,054-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 08:00:04,054-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:00:04,054-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:00:04,054-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3a45613d
>2018-05-30 08:00:04,054-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 08:00:04,068-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4587c15f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:00:04,069-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4587c15f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41a50f6b
>2018-05-30 08:00:04,071-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4587c15f] FINISH, SetVdsStatusVDSCommand, log id: 41a50f6b
>2018-05-30 08:00:04,080-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4587c15f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4587c15f, Job ID: ea7dccea-574b-4533-8c0c-a1338ea0f4a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 08:00:04,084-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4587c15f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 78bda7b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 08:00:04,087-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4587c15f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 78bda7b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 08:00:04,097-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [44b6282a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:00:04,098-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [44b6282a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 08:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2d185a67] Autorecovering 2 hosts
>2018-05-30 08:05:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2d185a67] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:05:00,043-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1e9934b6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 08:05:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1e9934b6] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 08:05:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1e9934b6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:05:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1e9934b6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:05:00,051-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1e9934b6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1c452a3b
>2018-05-30 08:05:00,054-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1e9934b6] FINISH, SetVdsStatusVDSCommand, log id: 1c452a3b
>2018-05-30 08:05:00,055-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1e9934b6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:05:00,057-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1e9934b6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 08:05:00,057-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [1e9934b6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:05:00,063-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [65aadced] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 08:05:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [65aadced] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 08:05:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [65aadced] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:05:00,069-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [65aadced] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:05:00,070-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [65aadced] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c74f196
>2018-05-30 08:05:00,072-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [65aadced] FINISH, SetVdsStatusVDSCommand, log id: 7c74f196
>2018-05-30 08:05:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [65aadced] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 500bf444
>2018-05-30 08:05:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [65aadced] FINISH, SetHaMaintenanceModeVDSCommand, log id: 500bf444
>2018-05-30 08:05:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [65aadced] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:05:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [65aadced] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 08:05:01,524-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [63ce6362] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 574e6ea
>2018-05-30 08:05:01,526-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [63ce6362] FINISH, GetHardwareInfoVDSCommand, log id: 574e6ea
>2018-05-30 08:05:01,539-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [23f8ec15] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:05:01,548-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [7d8b24f0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:05:01,553-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7e2029a3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:05:01,560-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7e2029a3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3a27699c
>2018-05-30 08:05:01,755-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3c48a448] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 46adf4a8
>2018-05-30 08:05:01,757-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3c48a448] FINISH, GetHardwareInfoVDSCommand, log id: 46adf4a8
>2018-05-30 08:05:01,769-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [5f9e17ee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:05:01,774-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [6cd8e277] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:05:01,779-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1ed1aef0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:05:01,786-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1ed1aef0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5f525406
>2018-05-30 08:05:01,941-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7e2029a3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 08:05:01,941-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7e2029a3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3a27699c
>2018-05-30 08:05:01,945-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7e2029a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:05:01,946-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7e2029a3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:05:01,949-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7e2029a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:05:01,950-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7e2029a3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:05:01,954-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7e2029a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:05:01,954-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7e2029a3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:05:01,957-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7e2029a3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 08:05:01,960-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7e2029a3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7e2029a3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 08:05:01,963-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6013e6f6
>2018-05-30 08:05:01,963-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 08:05:02,060-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:05:02,060-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 08:05:02,060-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:05:02,060-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:05:02,060-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6013e6f6
>2018-05-30 08:05:02,060-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 08:05:02,073-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [2180513b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:05:02,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2180513b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 552d5276
>2018-05-30 08:05:02,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2180513b] FINISH, SetVdsStatusVDSCommand, log id: 552d5276
>2018-05-30 08:05:02,108-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2180513b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2180513b, Job ID: 44df1017-7e1a-4919-aba5-7fa72d7b081f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 08:05:02,113-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2180513b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7d8b24f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 08:05:02,115-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2180513b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7d8b24f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 08:05:02,126-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [7c840a47] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:05:02,127-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [7c840a47] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 08:05:02,162-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1ed1aef0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 08:05:02,162-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1ed1aef0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5f525406
>2018-05-30 08:05:02,166-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1ed1aef0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:05:02,167-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1ed1aef0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:05:02,170-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1ed1aef0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:05:02,171-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1ed1aef0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:05:02,175-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1ed1aef0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:05:02,176-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1ed1aef0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:05:02,179-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1ed1aef0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 08:05:02,182-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1ed1aef0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1ed1aef0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 08:05:02,184-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: a3c4f3d
>2018-05-30 08:05:02,184-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 08:05:02,282-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:05:02,282-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 08:05:02,282-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:05:02,282-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:05:02,282-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: a3c4f3d
>2018-05-30 08:05:02,282-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 08:05:02,294-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [5f0f2dee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:05:02,296-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5f0f2dee] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5413e9ae
>2018-05-30 08:05:02,298-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5f0f2dee] FINISH, SetVdsStatusVDSCommand, log id: 5413e9ae
>2018-05-30 08:05:02,326-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5f0f2dee] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5f0f2dee, Job ID: 6f9e3dd2-58f3-4d2d-b0ca-443f765d7016, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 08:05:02,331-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5f0f2dee] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6cd8e277, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 08:05:02,333-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5f0f2dee] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6cd8e277, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 08:05:02,343-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [5662750c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:05:02,344-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [5662750c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 08:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7a5c65e8] Autorecovering 2 hosts
>2018-05-30 08:10:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7a5c65e8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:10:00,065-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1e29bdb0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 08:10:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1e29bdb0] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 08:10:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1e29bdb0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:10:00,072-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1e29bdb0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:10:00,074-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1e29bdb0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f6b8049
>2018-05-30 08:10:00,077-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1e29bdb0] FINISH, SetVdsStatusVDSCommand, log id: 7f6b8049
>2018-05-30 08:10:00,079-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1e29bdb0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:10:00,081-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1e29bdb0] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 08:10:00,081-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1e29bdb0] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:10:00,087-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2ba76] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 08:10:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2ba76] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 08:10:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2ba76] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:10:00,093-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2ba76] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:10:00,095-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2ba76] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47db49c6
>2018-05-30 08:10:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2ba76] FINISH, SetVdsStatusVDSCommand, log id: 47db49c6
>2018-05-30 08:10:00,099-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [2ba76] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: b193234
>2018-05-30 08:10:00,101-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [2ba76] FINISH, SetHaMaintenanceModeVDSCommand, log id: b193234
>2018-05-30 08:10:00,101-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2ba76] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:10:00,103-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2ba76] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 08:10:02,722-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [211d3074] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 28a49097
>2018-05-30 08:10:02,724-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [211d3074] FINISH, GetHardwareInfoVDSCommand, log id: 28a49097
>2018-05-30 08:10:02,738-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [1d180519] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:10:02,746-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [cba1e8e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:10:02,751-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [46994837] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:10:02,763-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [46994837] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6621658b
>2018-05-30 08:10:02,966-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [54a87bd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 366bf6a3
>2018-05-30 08:10:02,968-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [54a87bd] FINISH, GetHardwareInfoVDSCommand, log id: 366bf6a3
>2018-05-30 08:10:02,981-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [29d434ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:10:02,985-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [1b97d263] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:10:02,990-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [728a0802] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:10:02,997-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [728a0802] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 47371b5f
>2018-05-30 08:10:03,143-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [46994837] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 08:10:03,143-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [46994837] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6621658b
>2018-05-30 08:10:03,147-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [46994837] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:10:03,148-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [46994837] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:10:03,152-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [46994837] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:10:03,152-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [46994837] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:10:03,156-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [46994837] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:10:03,157-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [46994837] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:10:03,159-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [46994837] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 08:10:03,162-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [46994837] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 46994837, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 08:10:03,169-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 35ab692b
>2018-05-30 08:10:03,169-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 08:10:03,264-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:10:03,264-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 08:10:03,264-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:10:03,264-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:10:03,264-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 35ab692b
>2018-05-30 08:10:03,264-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 08:10:03,277-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [4145279d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:10:03,278-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4145279d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4a48617b
>2018-05-30 08:10:03,280-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4145279d] FINISH, SetVdsStatusVDSCommand, log id: 4a48617b
>2018-05-30 08:10:03,312-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4145279d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4145279d, Job ID: e8588bc1-2e3d-4c27-bfbb-7cb5bbdd029b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 08:10:03,317-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4145279d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: cba1e8e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 08:10:03,319-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4145279d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: cba1e8e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 08:10:03,331-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [13710997] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:10:03,333-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [13710997] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 08:10:04,364-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [728a0802] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 08:10:04,364-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [728a0802] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 47371b5f
>2018-05-30 08:10:04,368-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [728a0802] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:10:04,369-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [728a0802] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:10:04,372-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [728a0802] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:10:04,373-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [728a0802] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:10:04,377-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [728a0802] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:10:04,378-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [728a0802] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:10:04,381-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [728a0802] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 08:10:04,383-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [728a0802] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 728a0802, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 08:10:04,386-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 21a99e9
>2018-05-30 08:10:04,386-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 08:10:04,482-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:10:04,482-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 08:10:04,482-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:10:04,482-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:10:04,482-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 21a99e9
>2018-05-30 08:10:04,482-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 08:10:04,495-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [46af242a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:10:04,496-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [46af242a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b62afdb
>2018-05-30 08:10:04,498-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [46af242a] FINISH, SetVdsStatusVDSCommand, log id: b62afdb
>2018-05-30 08:10:04,504-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [46af242a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 46af242a, Job ID: aa2f21ba-82ff-45b0-9538-da6322e8cf49, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 08:10:04,508-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [46af242a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1b97d263, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 08:10:04,510-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [46af242a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1b97d263, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 08:10:04,521-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [46e0878a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:10:04,522-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [46e0878a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 08:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [519a5421] Autorecovering 2 hosts
>2018-05-30 08:15:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [519a5421] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:15:00,064-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [488f5eca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 08:15:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [488f5eca] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 08:15:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [488f5eca] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:15:00,071-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [488f5eca] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:15:00,073-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [488f5eca] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46c4e30c
>2018-05-30 08:15:00,075-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [488f5eca] FINISH, SetVdsStatusVDSCommand, log id: 46c4e30c
>2018-05-30 08:15:00,077-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [488f5eca] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:15:00,078-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [488f5eca] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 08:15:00,079-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [488f5eca] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:15:00,084-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3bac8d13] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 08:15:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3bac8d13] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 08:15:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3bac8d13] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:15:00,090-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3bac8d13] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:15:00,091-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3bac8d13] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3c83c290
>2018-05-30 08:15:00,093-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3bac8d13] FINISH, SetVdsStatusVDSCommand, log id: 3c83c290
>2018-05-30 08:15:00,096-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [3bac8d13] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 34b88ac6
>2018-05-30 08:15:00,097-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [3bac8d13] FINISH, SetHaMaintenanceModeVDSCommand, log id: 34b88ac6
>2018-05-30 08:15:00,098-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3bac8d13] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:15:00,099-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3bac8d13] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 08:15:01,002-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [67fb7d16] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 269997f4
>2018-05-30 08:15:01,003-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [67fb7d16] FINISH, GetHardwareInfoVDSCommand, log id: 269997f4
>2018-05-30 08:15:01,016-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [66598c66] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:15:01,024-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [3acebfa1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:15:01,029-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1c2bb673] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:15:01,041-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1c2bb673] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1df7c27
>2018-05-30 08:15:01,423-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1c2bb673] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 08:15:01,423-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1c2bb673] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1df7c27
>2018-05-30 08:15:01,427-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1c2bb673] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:15:01,428-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1c2bb673] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:15:01,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1c2bb673] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:15:01,432-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1c2bb673] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:15:01,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1c2bb673] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:15:01,437-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1c2bb673] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:15:01,439-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1c2bb673] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 08:15:01,442-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1c2bb673] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1c2bb673, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 08:15:01,448-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6b945bd1
>2018-05-30 08:15:01,448-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 08:15:01,547-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:15:01,547-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 08:15:01,547-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:15:01,547-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:15:01,547-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6b945bd1
>2018-05-30 08:15:01,547-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 08:15:01,560-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [3b7fa798] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:15:01,561-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3b7fa798] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5d2e6a8
>2018-05-30 08:15:01,563-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3b7fa798] FINISH, SetVdsStatusVDSCommand, log id: 5d2e6a8
>2018-05-30 08:15:01,593-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3b7fa798] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3b7fa798, Job ID: 54ef9017-0ab6-4f07-a5e9-17928fae9ef1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 08:15:01,598-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3b7fa798] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3acebfa1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 08:15:01,600-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3b7fa798] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3acebfa1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 08:15:01,611-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [28d88cee] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:15:01,612-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [28d88cee] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 08:15:02,251-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7be1a1be] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6433d906
>2018-05-30 08:15:02,253-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7be1a1be] FINISH, GetHardwareInfoVDSCommand, log id: 6433d906
>2018-05-30 08:15:02,265-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [2befad20] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:15:02,270-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [51db8121] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:15:02,274-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4dc4432f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:15:02,281-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4dc4432f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6ca2b71b
>2018-05-30 08:15:02,657-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4dc4432f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 08:15:02,658-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4dc4432f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6ca2b71b
>2018-05-30 08:15:02,661-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4dc4432f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:15:02,662-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4dc4432f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:15:02,666-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4dc4432f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:15:02,666-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4dc4432f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:15:02,670-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4dc4432f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:15:02,671-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4dc4432f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:15:02,674-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4dc4432f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 08:15:02,677-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4dc4432f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4dc4432f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 08:15:02,683-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 283fb76e
>2018-05-30 08:15:02,683-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 08:15:02,783-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:15:02,783-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 08:15:02,783-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:15:02,783-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:15:02,783-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 283fb76e
>2018-05-30 08:15:02,783-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 08:15:02,796-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [14ab19cd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:15:02,797-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [14ab19cd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 719abeb0
>2018-05-30 08:15:02,799-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [14ab19cd] FINISH, SetVdsStatusVDSCommand, log id: 719abeb0
>2018-05-30 08:15:02,831-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [14ab19cd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 14ab19cd, Job ID: 84005ed5-b352-44ba-a916-08ba95ae7775, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 08:15:02,835-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [14ab19cd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 51db8121, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 08:15:02,837-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [14ab19cd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 51db8121, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 08:15:02,848-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [aa3585f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:15:02,849-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [aa3585f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 08:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6cf0fda1] Autorecovering 2 hosts
>2018-05-30 08:20:00,009-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6cf0fda1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:20:00,016-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3b45752a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 08:20:00,022-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3b45752a] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 08:20:00,022-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3b45752a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:20:00,022-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3b45752a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:20:00,024-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3b45752a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2140b73f
>2018-05-30 08:20:00,026-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3b45752a] FINISH, SetVdsStatusVDSCommand, log id: 2140b73f
>2018-05-30 08:20:00,028-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3b45752a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:20:00,030-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3b45752a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 08:20:00,030-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3b45752a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:20:00,035-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5951f10e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 08:20:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5951f10e] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 08:20:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5951f10e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:20:00,041-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5951f10e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:20:00,043-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5951f10e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4db9391d
>2018-05-30 08:20:00,045-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5951f10e] FINISH, SetVdsStatusVDSCommand, log id: 4db9391d
>2018-05-30 08:20:00,048-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5951f10e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5ec8f68a
>2018-05-30 08:20:00,050-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5951f10e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5ec8f68a
>2018-05-30 08:20:00,050-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5951f10e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:20:00,051-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5951f10e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 08:20:00,475-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6512b5a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1ad7bd68
>2018-05-30 08:20:00,477-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6512b5a] FINISH, GetHardwareInfoVDSCommand, log id: 1ad7bd68
>2018-05-30 08:20:00,490-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [27b82650] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:20:00,498-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1671bd87] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:20:00,503-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [200e3f43] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:20:00,509-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [200e3f43] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 65596f02
>2018-05-30 08:20:00,880-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [200e3f43] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 08:20:00,880-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [200e3f43] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 65596f02
>2018-05-30 08:20:00,884-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [200e3f43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:20:00,885-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [200e3f43] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:20:00,888-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [200e3f43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:20:00,889-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [200e3f43] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:20:00,893-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [200e3f43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:20:00,894-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [200e3f43] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:20:00,897-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [200e3f43] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 08:20:00,899-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [200e3f43] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 200e3f43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 08:20:00,902-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2f732833
>2018-05-30 08:20:00,902-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 08:20:01,001-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:20:01,001-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 08:20:01,001-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:20:01,001-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:20:01,001-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2f732833
>2018-05-30 08:20:01,001-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 08:20:01,013-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [21fa65c2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:20:01,015-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [21fa65c2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f5ac6fe
>2018-05-30 08:20:01,017-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [21fa65c2] FINISH, SetVdsStatusVDSCommand, log id: 3f5ac6fe
>2018-05-30 08:20:01,022-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [21fa65c2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 21fa65c2, Job ID: efff871a-6688-4531-a860-10cc5f5c9b95, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 08:20:01,026-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [21fa65c2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1671bd87, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 08:20:01,028-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [21fa65c2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1671bd87, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 08:20:01,039-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [6cfa7905] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:20:01,040-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [6cfa7905] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 08:20:02,247-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [705aae7b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 35ab33d1
>2018-05-30 08:20:02,249-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [705aae7b] FINISH, GetHardwareInfoVDSCommand, log id: 35ab33d1
>2018-05-30 08:20:02,263-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [7dbd4ee9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:20:02,270-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [2580198e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:20:02,276-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [64777bc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:20:02,285-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [64777bc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2705d56a
>2018-05-30 08:20:02,663-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [64777bc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 08:20:02,663-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [64777bc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2705d56a
>2018-05-30 08:20:02,666-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [64777bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:20:02,667-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [64777bc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:20:02,671-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [64777bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:20:02,672-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [64777bc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:20:02,675-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [64777bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:20:02,676-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [64777bc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:20:02,680-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [64777bc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 08:20:02,682-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [64777bc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 64777bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 08:20:02,685-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: d7fd559
>2018-05-30 08:20:02,685-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 08:20:02,783-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:20:02,783-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 08:20:02,783-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:20:02,783-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:20:02,783-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: d7fd559
>2018-05-30 08:20:02,783-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 08:20:02,796-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [4d6ff912] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:20:02,798-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4d6ff912] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72ec12c2
>2018-05-30 08:20:02,800-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4d6ff912] FINISH, SetVdsStatusVDSCommand, log id: 72ec12c2
>2018-05-30 08:20:02,805-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4d6ff912] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4d6ff912, Job ID: 4d22251e-5044-4aeb-b78d-d0e8e05895b1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 08:20:02,810-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4d6ff912] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2580198e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 08:20:02,812-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4d6ff912] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2580198e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 08:20:02,823-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [62606c18] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:20:02,824-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [62606c18] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 08:20:24,890-04 INFO  [org.ovirt.engine.extension.aaa.jdbc.core.Tasks] (default task-9) [] (house keeping) deleting failed logins prior to 2018-05-23 12:20:24Z.
>2018-05-30 08:20:24,912-04 INFO  [org.ovirt.engine.extension.aaa.jdbc.core.Tasks] (default task-9) [] (house keeping) deleting failed logins prior to 2018-05-23 12:20:24Z.
>2018-05-30 08:20:24,924-04 INFO  [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-9) [] User admin@internal successfully logged in with scopes: ovirt-app-admin ovirt-app-api ovirt-app-portal ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access
>2018-05-30 08:20:25,070-04 INFO  [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-20) [63ca92bc] Running command: CreateUserSessionCommand internal: false.
>2018-05-30 08:20:25,077-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-20) [63ca92bc] EVENT_ID: USER_VDC_LOGIN(30), Correlation ID: 63ca92bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User admin@internal-authz logged in.
>2018-05-30 08:22:13,188-04 INFO  [org.ovirt.engine.core.bll.provider.AddProviderCommand] (default task-8) [51ebbe28-4fd0-4306-b459-4cbdd340000b] Running command: AddProviderCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_POOL with role type ADMIN
>2018-05-30 08:22:13,211-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-8) [51ebbe28-4fd0-4306-b459-4cbdd340000b] EVENT_ID: PROVIDER_ADDED(205), Correlation ID: 51ebbe28-4fd0-4306-b459-4cbdd340000b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Provider OSP_Glance was added. (User: admin@internal-authz)
>2018-05-30 08:25:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3be005f5] Autorecovering 2 hosts
>2018-05-30 08:25:00,005-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3be005f5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:25:00,033-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [365c2772] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 08:25:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [365c2772] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 08:25:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [365c2772] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:25:00,040-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [365c2772] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:25:00,042-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [365c2772] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4dc464ae
>2018-05-30 08:25:00,044-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [365c2772] FINISH, SetVdsStatusVDSCommand, log id: 4dc464ae
>2018-05-30 08:25:00,045-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [365c2772] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:25:00,047-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [365c2772] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}'
>2018-05-30 08:25:00,047-04 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [365c2772] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:25:00,052-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76057a69] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 08:25:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76057a69] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN
>2018-05-30 08:25:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76057a69] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:25:00,058-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76057a69] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:25:00,060-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [76057a69] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5cb10eee
>2018-05-30 08:25:00,062-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [76057a69] FINISH, SetVdsStatusVDSCommand, log id: 5cb10eee
>2018-05-30 08:25:00,064-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [76057a69] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: a57c914
>2018-05-30 08:25:00,067-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [76057a69] FINISH, SetHaMaintenanceModeVDSCommand, log id: a57c914
>2018-05-30 08:25:00,067-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76057a69] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default'
>2018-05-30 08:25:00,068-04 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76057a69] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}'
>2018-05-30 08:25:00,394-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [7fc5d6bf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 57660bb5
>2018-05-30 08:25:00,396-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [7fc5d6bf] FINISH, GetHardwareInfoVDSCommand, log id: 57660bb5
>2018-05-30 08:25:00,408-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [19a23852] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:25:00,413-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [4b15f298] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:25:00,418-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7bb21484] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:25:00,430-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7bb21484] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1a4ed492
>2018-05-30 08:25:00,814-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7bb21484] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 08:25:00,814-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7bb21484] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1a4ed492
>2018-05-30 08:25:00,818-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7bb21484] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:25:00,819-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7bb21484] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:25:00,823-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7bb21484] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:25:00,823-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7bb21484] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:25:00,827-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7bb21484] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:25:00,828-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7bb21484] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:25:00,830-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7bb21484] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 08:25:00,833-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7bb21484] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7bb21484, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 08:25:00,840-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 751c45ad
>2018-05-30 08:25:00,840-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 08:25:00,934-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:25:00,934-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 08:25:00,934-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:25:00,934-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:25:00,934-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 751c45ad
>2018-05-30 08:25:00,934-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 08:25:00,947-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [328a3d4b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:25:00,948-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [328a3d4b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 539c6dd5
>2018-05-30 08:25:00,950-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [328a3d4b] FINISH, SetVdsStatusVDSCommand, log id: 539c6dd5
>2018-05-30 08:25:00,959-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [328a3d4b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 328a3d4b, Job ID: b5951458-a9a8-4b6e-acc7-40a1dd72b2f8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 08:25:00,972-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [328a3d4b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4b15f298, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com.
>2018-05-30 08:25:00,974-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [328a3d4b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4b15f298, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 08:25:00,985-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [ed82ca1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS
>2018-05-30 08:25:00,986-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [ed82ca1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
>2018-05-30 08:25:01,652-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [39d26858] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4bb0c3ed
>2018-05-30 08:25:01,654-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [39d26858] FINISH, GetHardwareInfoVDSCommand, log id: 4bb0c3ed
>2018-05-30 08:25:01,666-04 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [5ab51c4b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:25:01,671-04 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [1b7f61c8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:25:01,676-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4cad2a69] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool
>2018-05-30 08:25:01,683-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4cad2a69] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5e4632e4
>2018-05-30 08:25:02,049-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4cad2a69] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage.
>2018-05-30 08:25:02,049-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4cad2a69] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5e4632e4
>2018-05-30 08:25:02,053-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4cad2a69] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:25:02,054-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4cad2a69] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:25:02,057-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4cad2a69] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:25:02,058-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4cad2a69] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:25:02,062-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4cad2a69] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target
>2018-05-30 08:25:02,062-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4cad2a69] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target
>2018-05-30 08:25:02,065-04 INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4cad2a69] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed 
>2018-05-30 08:25:02,067-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4cad2a69] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4cad2a69, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers
>2018-05-30 08:25:02,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7219c14b
>2018-05-30 08:25:02,070-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active}
>2018-05-30 08:25:02,166-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:25:02,166-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]'
>2018-05-30 08:25:02,166-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com
>2018-05-30 08:25:02,166-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5'
>2018-05-30 08:25:02,166-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7219c14b
>2018-05-30 08:25:02,166-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null
>2018-05-30 08:25:02,179-04 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [f822199] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:25:02,180-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [f822199] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 38fa16f5
>2018-05-30 08:25:02,182-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [f822199] FINISH, SetVdsStatusVDSCommand, log id: 38fa16f5
>2018-05-30 08:25:02,188-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [f822199] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: f822199, Job ID: f8f93842-36f8-4e9f-a4b2-538565aa6793, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
>2018-05-30 08:25:02,192-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [f822199] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1b7f61c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com.
>2018-05-30 08:25:02,195-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [f822199] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1b7f61c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default
>2018-05-30 08:25:02,205-04 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [746e7f5b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS
>2018-05-30 08:25:02,206-04 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [746e7f5b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.

Actions: View

Attachments on bug 1581309: 1440167 | 1440169 | 1440170 | 1442700 | 1442701 | 1445827